Install GO on linux


Go is high level scripting language developed at Google by by Robert Griesemer, Rob Pike, and Ken Thompson in 2007.It is an open source programming language. It's a fast, statically typed, compiled language that feels like a dynamically typed, interpreted language.In this article we will learn installation of GO on Ubuntu.We also try for first program "Hello World" in go.

Installation : Update the repo:
$sudo apt-get update

You can install in two ways as follows: Install from CLI, It will install latest version of GO:
$sudo apt-get install golang


 Downlaod specifi version of GO language by downloading package:
$sudo curl -O

Extract Downloaded package:
$sudo tar -xvf go1.6.linux-amd64.tar.gz

move to /usr/local
$sudo mv go /usr/local

Setting Go Paths:
$vim ~/.bashrc

add the following lines at the end:
export PATH=$PATH:/usr/local/go/bin

Apply the changes:
$source ~/.bashrc

Check Installation of GO:
$ go
Go is a tool for managing Go source code.


        go command [arguments]

The commands are:

        build       compile packages and dependencies
        clean       remove object files
        doc         show documentation for package or symbol
        env         print Go environment information
        fix         run go tool fix on packages
        fmt         run gofmt on package sources
        generate    generate Go files by processing source
        get         download and install packages and dependencies
        install     compile and install packages and dependencies
        list        list packages
        run         compile and run Go program
        test        test packages
        tool        run specified go tool
        version     print Go version
        vet         run go tool vet on packages

Use "go help [command]" for more information about a command.

Additional help topics:

        c           calling between Go and C
        buildmode   description of build modes
        filetype    file types
        gopath      GOPATH environment variable
        environment environment variables
        importpath  import path syntax
        packages    description of package lists
        testflag    description of testing flags
        testfunc    description of testing functions

Use "go help [topic]" for more information about that topic.

Lest try Hello World in GO:

I am using my favourite editor vim, you can use any editor.
$vim Hello.go

package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")

$ hello.go
hello, world

Install hadoop-2.7.3 on Ubuntu 16.04


In this article we are installing hadoop-2.7.3 on single node.Single system is sufficient to run all components of hadoop. But at production level hadoop cluster involves many more machines. Hadoop runs on any linux ditribution.

Install Java:

Hadoop is written in Java, so it requires Java Development Kit(JDK) on your system.
Check java is available on your system by following command:

$ javac
$ java -version

If either of this commands gives error then you have to install java first.
Steps to install java :

$ sudo apt-get update
$ sudo apt-get install openjdk-8-jre
$ sudo apt-get install openjdk-8-jdk
$ java -version

Once java is installed set JAVA_HOME
environmental variable in .bashrc file.
Before that how to find where java is installed? use following commands:

$ ls -l /etc/alternatives/javac
lrwxrwxrwx 1 root root 36 Nov 14 23:15 /etc/alternatives/javac -> /usr/lib/jvm/java-8-oracle/bin/javac

/usr/lib/jvm/java-8-oracle this is the location where java is installed.

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin

Now copy above two lines and paste at end in .bashrc file. Your java path may be different.

Configure ssh

Hadoop requires communication between multiple components present on one or more machines. We need to ensure that user we are using for hadoop can connect to required host without needing password. It can be done by using SSH
If ssh is not available on your system then install using following command:

$ sudo apt-get install ssh

After installation of ssh execute following commands:

$ ssh-keygen -t rsa -P ""
$ cat $HOME/.ssh/ >> $HOME/.ssh/authorized_keys
$ ssh localhost

Download hadoop

To dowanload hadoop click here.

Extract and move hadoop-2.7.3 folder to /usr/local/

$ mv hadoop-2.7.3 /usr/local

Configure .bashrc file

export HADOOP_HOME=/usr/local/hadoop-2.7.3
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Update .bashrc file to apply changes

$ source ~/.bashrc



Open file and find export JAVA_HOME and set JAVA_HOME path.

$ vim /usr/local/hadoop-2.7.3/etc/hadoop/
export JAVA_HOME=/usr/lib/jvm/java-8-oracle

Modify core-site.xml

$ vim /usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml

Modify hdfs-site.xml

$ vim /usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml

Modify mapred-site.xml

  $ vim /usr/local/hadoop-2.7.3/etc/hadoop/mapred-site.xml

Modify yarn-site.xml

 $ vim /usr/local/hadoop-2.7.3/etc/hadoop/yarn-site.xml

Format namenode for first time

$ hadoop namenode -format

Start hadoop


Check proper installation of hadoop

 $ jps
6482 DataNode
5763 ResourceManager
6694 SecondaryNameNode
6937 NodeManager
7182 Jps
6367 NameNode

If all 5 commands are running properly means your hadoop is working properly. :)

Browse the web interface for the NameNode, ResourceManager

NameNode - http://localhost:50070/

SecondaryNameNode - http://localhost:50090/status.html

DataNode - http://localhost:50070/dfshealth.html#tab-datanode

ResourceManager - http://localhost:8088/

Stop Hadoop:

Thats all in this article, we learned about running hadoop locally.
Start exploring your interset in hadoop.

Dont forget to like and share this post!

Install Devstack on Linux


DevStack lets you interact with openstack on a small scale that’s representative of a much larger deployment. Devstack helps you deploy the same openstack components found in large multi-server environments on a single server. You can quickly deploy devstack. Insted of deploying Controller and compute node separately on different VMs/system we use sing VMs/system to run openstack, and its nothing but devstack.

In this article we are installing devstack on Ubuntu 16.04.
You can install devstack on any Linux distribution(Ubuntu/CentOS/RHEL)

Install devstack:
Update local packages:

# sudo apt-get -y update

Install git

# sudo apt-get -y install git

Using git, retrieve the latest release of devStack to /opt/devsatck directory:

# sudo git clone /opt/devstack/

DON’T STACK AS ROOT. If you try to stack as the root user with devStack, the process will fail with an error scolding you for running the script as the root user.

Enter Into devstack directory:

# cd /opt/devstack/

Create the stack user and set ownership of all devStack files to that user:

# sudo chmod u+x tools/
# sudo tools/
Creating a group called stack
Creating a user called stack
Giving stack user passwordless sudo privileges

Makes stack user owner of all files in directory:

# sudo chown -R stack:stack /opt/devstack/

Your directory has now been prepared with appropriate permissions, and a new user has been created.

Switches to stack user

# sudo -i -u stack

Go into /devstack directory:

# cd /opt/devstack/

Creating local.conf in /devstack directory. You’ll build your local.conf file, which is used by devStack to configure your deployment

# vim local.conf

# Credentials

# Output


# ./

Go ahead your devstack is installed successfully. Access the Dash-board by entering the following URL into your browser:

http://<your host_ip>

Enter the following user name and password:

User name : admin/demo
Password : test

 Lets start exploring devstack!

This is how instances in devstack will look like:
I have created two instances inst1 and my_test you can see details of any instance by just clicking on instance name.

Restart devstack:

After restart of your machine we want our devstack in last session state then use following commands:
# screen -c stack-screenrc

Remember :There may chance of error like:
Cannot open your terminal '/dev/pts/7' - please check. Means you require root privileges. So run above command using root.

Uninstall devstack:

How to remove devstack from system:
Its very simple just enter following command:

# ./
# ./
# sudo rm -rf /opt/stack
# sudo reboot

That's all in this article. Don't forget to share your views in comment section below.

basic networking commands in linux


Linux system is command-based. To enjoy real beauty of Linux, you must aware with commands used in Linux. All commands are located in /bin and /usr/bin. In man command contains documentation of commands. In this article we learn few basic networking commands like host-name, ifconfig, ping and so many. Lets start with commands:

It shows the systems Hostname. Hostname file present at "/etc/hostname" location. We can change host name permanently by editing this file.

# hostname

IFCONFIG command
ifconfig (interface configurator) displays ip address, netmask, brodcast network interface, hardware address and current network configuration information.

# ifconfig
enp19s0   Link encap:Ethernet  HWaddr a4:ba:db:xx:xx:xx  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:  Mask:
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2204 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2204 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:183514 (183.5 KB)  TX bytes:183514 (183.5 KB)

wlp18s0b1 Link encap:Ethernet  HWaddr c4:46:19:xx:xx:xx  
          inet addr:192.168.x.x  Bcast:192.168.x.xx  Mask:
          inet6 addr: fe80::22f3:f1e1:xxxx:xxxx/64 Scope:Link
          RX packets:18627 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11016 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:20908110 (20.9 MB)  TX bytes:1289043 (1.2 MB)
ifconfig with interface name (enp19s0) will display only specific interface details.
we can enable specific interface using commands "ifup interface_name" or disable specific interface using commands "ifdown interface_name"

# ifup enp19s0

# ifdown enp19s0

PING command
Ping is used to test the network.It send ICMP ECHO_REQUEST to network hosts.

ping using domain name:
# ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=46 time=66.3 ms
64 bytes from ( icmp_seq=2 ttl=46 time=72.3 ms
64 bytes from ( icmp_seq=3 ttl=46 time=63.3 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 63.313/67.306/72.301/3.743 ms

ping using IP address:
# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=46 time=63.3 ms
64 bytes from icmp_seq=2 ttl=46 time=62.7 ms
64 bytes from icmp_seq=3 ttl=46 time=62.4 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 62.416/62.833/63.363/0.394 ms

DIG command
dig (domain information groper) is a flexible tool for interrogating DNS name servers. It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried.


; <<>> DiG 9.10.3-P4-Ubuntu <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- 06:42:16="" 0="""" 14321="" 16759="" 177="" 1799="" 183="" 1="" 2016="""" 30="" 3="" 4096="" 472="" 4="" 97977="" a="" additional:="" answer:="" answer="" authority:="" authority="" cname="" edns:="" flags:="""""""" id:="" in="" ist="" msec="" msg="" noerror="" nov="""""""""" ns="" opcode:="" opt="" pre="" pseudosection:="" qr="" query:="" query="" question="" ra="" rcvd:="" rd="" section:="" server:="" size="" status:="" time:="" udp:="" version:="" wed="" when:="""">

NETSTAT command
This command print network connections, routing tables, interface statistics, masquerade connections, and multi-cast memberships. Useful for finding connection to and from the host.
netstat -t : display only tcp connection
netstat -u : display only udp connection
netstat -g : display all interfaces
netstat -a :display all connection
netstat -nap | grep port : display pid of application using this port
netstat -l : List only listing port

# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 192.168.x.x:xxxxx       ec2-35-xxx-xx-xxx:https ESTABLISHED
tcp        0      0 192.168.x.x:xxxxx    ESTABLISHED
tcp        0      0 localhost:epmd          localhost:49290         TIME_WAIT  
tcp        0      0 localhost:58746         localhost:epmd          ESTABLISHED
tcp        0      0 localhost:epmd          localhost:58746         ESTABLISHED
tcp        0      0 192.168.x.x:xxxxx       bom05s05-in-f14.1:https ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ]         DGRAM                    21494    /run/user/1000/systemd/notify
unix  3      [ ]         DGRAM                    11869    /run/systemd/notify
unix  7      [ ]         DGRAM                    11880    /run/systemd/journal/socket
unix  2      [ ]         DGRAM                    11888    /run/systemd/journal/syslog

NSLOOKUP command
It discovers Hostname from IP address or discover IP address from hostname.

# nslookup

Non-authoritative answer:
# nslookup

Non-authoritative answer: name =

Authoritative answers can be found from: nameserver = nameserver = nameserver = nameserver =

ROUTE command
It shows ip routing table. We can also manipulate this table.

# route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         UG    600    0        0 wlp18s0b1
link-local      *          U     1000   0        0 wlp18s0b1     *        U     600    0        0 wlp18s0b1
We can add/delete route.
Route add

# route add -net  gw 

Route Delete

# route del -net  gw 

traceroute : IT tracks the route packets taken from an IP network on their way to a given host. It utilizes the IP protocol's time to live (TTL) field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path to the host.

# traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  2.113 ms  2.814 ms  3.600 ms
 2  * * *
 3 (  17.259 ms  18.846 ms  20.375 ms
 4  * * *
 5  * * *
 6 (  29.267 ms * *
 7 (  20.147 ms  15.656 ms  14.705 ms
 8 (  17.392 ms  8.787 ms  11.068 ms
 9 (  10.239 ms  9.947 ms  8.268 ms
10  * * *
11  * * *
12  * * *
13 (  17.852 ms  19.361 ms  21.089 ms

Start enjoying Linux!

Linux commands for Hadoop

The file is nothing but storage of information. On Linux or Unix system everything is considered as File. So system contains a large number of files. All details of files are stored in a separate area of hard disk which is accessible only to the kernel.

The file is mainly divided into 3 categories:
  1. Ordinary File : Regular files contain only data as a stream of character. e.g. Text files, Binary files.
  2.  Directory File : Contains file and other directory. (It contains file name and corresponding inode number)
  3.  Device File : All devices and peripherals are represented by files. Generally, files found under /dev directory are device files.
/dev - all device file content.
/etc - all configuration files of the system.
/bin & /usr/bin - all commonly used linux commands.
/sbin & /usr/sbin - System administrator can execute these commands.
/usr/include - contains header files used by compilers.
/lib & /usr/lib - contain all library files in binary form.
/tmp - user can create temporary files.
/var - contains variable part of the file system.   
/home - home directory.
Home directory
kb@kb:$ echo $HOME
kb@kb: ~$

Move from any directory to home directory: $ cd ~
Move from any directory to root directory :  $ cd /
Checking your current directory:(pwd)
kb@kb: ~/Desktop$ pwd
kb@kb: ~/Desktop$
Changing your current directory (cd):
kb@kb: ~/Desktop$ pwd
kb@kb: ~/Desktop$ cd Image/
kb@kb: ~/Desktop/Image$ pwd

Current directory: "Desktop/" Change directory to "/Image":
$ cd Image/ Now current directory: Image/
Making directory (mkdir):
Create a new directory inside Image directory: 
$ mkdir Data

kb@kb: ~/Desktop/Image$ mkdir Data
kb@kb: ~/Desktop/Image/Data

Create number of directories in another directory using command :
$ mkdir -P Data/Hadoop/LinuxCommands
Removing directory/file (rm):
Remove the directory Data from Image directory : 
$ rmdir Data

kb@kb: ~/Desktop/Image$ mkdir Data
kb@kb: ~/Desktop/Image$ ls
Actor Cute Friends    Nature Data God 
kb@kb: ~/Desktop/Image$ rmdir Data
kb@kb: ~/Desktop/Image$ ls
Actor Cute Friends    Nature God

Removing directories/file from another directory using command :
$ rm -r Data/Hadoop/LinuxCommands   (r - recursively)
$ rm * (Delete all files)
$ rm file1 file2 file3 (Deleting file1 file2 file3 )
Absolute pathnames:
Complete path name is mentioned in terminal. 
Run Linux command directly by mentioning the path: e.g. /bin/date
Relative pathnames:
Mention path from the current directory if the file is present in that directory. 
. (single dot) - This represents the current directory
.. (two dots) - This represents the parent directory
e.g. $ pwd
$ cd .. (change directory to parent of the current directory)
$ pwd
$ cd ../.. (change directory to parent of parent of current directory)
Listing directory content (ls):
ls       - list of all filename in the current directory .
ls -x   - Multi columnar output
ls -a   - Show all filename beginning with a dot including . and .. (Header File)
ls -R  - Recursive list
ls - r  - Sort filename in reverse order 
ls -l   - all details about files and directories  
           If starting with "-" Its a file.
           If starting with "d" Its a directory
ls -t    - Sort filenames by last modification time.
Copying a File (cp):
cp command copies a file or a group of files.
$ cp  
$ cp Demo/* /Demo/HDFS (Copy all files from Demo to HDFS dir) 
Renaming Files (mv) :
It rename a file/dir or moves a group of file to different dir.
$ mv file1 file2
Creating and Displaying file:
$ cat > filename (create a file)
$ cat >> filename (create file if not exist and add content,
                if file exist then append content) 
$ cat filename (show content of file)
To create file generally used techniques:
          1) Vi editor
          2) touch command
          3) gedit editor 
Creating and Extracting Jar files:
$ jar -cvf "filename"(.jar) "src"(.java)  //(cvf - create verbos file)
$ jar -cvf wc

Extract jar:
$ jar -xvf filename.jar 
Change File Permission (chmod):
$ chmod 755 file.txt

7 r w x (root user)
5 r - x (group user)
5 r - x (other user)
There are plenty of commands in linux, but we require only few of them listed above enough to work with hadoop. After practicing above commands, you have enough knowledge of linux command to start working on hadoop.