Linux commands for Hadoop

The file is nothing but storage of information. On Linux or Unix system everything is considered as File. So system contains a large number of files. All details of files are stored in a separate area of hard disk which is accessible only to the kernel.

The file is mainly divided into 3 categories:
  1. Ordinary File : Regular files contain only data as a stream of character. e.g. Text files, Binary files.
  2.  Directory File : Contains file and other directory. (It contains file name and corresponding inode number)
  3.  Device File : All devices and peripherals are represented by files. Generally, files found under /dev directory are device files.
/dev - all device file content.
/etc - all configuration files of the system.
/bin & /usr/bin - all commonly used linux commands.
/sbin & /usr/sbin - System administrator can execute these commands.
/usr/include - contains header files used by compilers.
/lib & /usr/lib - contain all library files in binary form.
/tmp - user can create temporary files.
/var - contains variable part of the file system.   
/home - home directory.
Home directory
kb@kb:$ echo $HOME
/home/kb
kb@kb: ~$

Move from any directory to home directory: $ cd ~
Move from any directory to root directory :  $ cd /
Checking your current directory:(pwd)
kb@kb: ~/Desktop$ pwd
/home/kb/Desktop
kb@kb: ~/Desktop$
Changing your current directory (cd):
kb@kb: ~/Desktop$ pwd
/home/kb/Desktop
kb@kb: ~/Desktop$ cd Image/
kb@kb: ~/Desktop/Image$ pwd
/home/kb/Desktop/Image/

Current directory: "Desktop/" Change directory to "/Image":
$ cd Image/ Now current directory: Image/
Making directory (mkdir):
Create a new directory inside Image directory: 
$ mkdir Data

kb@kb: ~/Desktop/Image$ mkdir Data
kb@kb: ~/Desktop/Image/Data

Create number of directories in another directory using command :
$ mkdir -P Data/Hadoop/LinuxCommands
Removing directory/file (rm):
Remove the directory Data from Image directory : 
$ rmdir Data

kb@kb: ~/Desktop/Image$ mkdir Data
kb@kb: ~/Desktop/Image$ ls
Actor Cute Friends    Nature Data God 
kb@kb: ~/Desktop/Image$ rmdir Data
kb@kb: ~/Desktop/Image$ ls
Actor Cute Friends    Nature God

Removing directories/file from another directory using command :
$ rm -r Data/Hadoop/LinuxCommands   (r - recursively)
$ rm * (Delete all files)
$ rm file1 file2 file3 (Deleting file1 file2 file3 )
Absolute pathnames:
Complete path name is mentioned in terminal. 
Run Linux command directly by mentioning the path: e.g. /bin/date
Relative pathnames:
Mention path from the current directory if the file is present in that directory. 
. (single dot) - This represents the current directory
.. (two dots) - This represents the parent directory
e.g. $ pwd
/home/kb/Desktop/Image/
$ cd .. (change directory to parent of the current directory)
$ pwd
/home/kb/Desktop
$ cd ../.. (change directory to parent of parent of current directory)
/home
Listing directory content (ls):
ls       - list of all filename in the current directory .
ls -x   - Multi columnar output
ls -a   - Show all filename beginning with a dot including . and .. (Header File)
ls -R  - Recursive list
ls - r  - Sort filename in reverse order 
ls -l   - all details about files and directories  
           If starting with "-" Its a file.
           If starting with "d" Its a directory
ls -t    - Sort filenames by last modification time.
Copying a File (cp):
cp command copies a file or a group of files.
$ cp  
$ cp Demo/* /Demo/HDFS (Copy all files from Demo to HDFS dir) 
Renaming Files (mv) :
It rename a file/dir or moves a group of file to different dir.
$ mv file1 file2
Creating and Displaying file:
$ cat > filename (create a file)
$ cat >> filename (create file if not exist and add content,
                if file exist then append content) 
$ cat filename (show content of file)
To create file generally used techniques:
          1) Vi editor
          2) touch command
          3) gedit editor 
Creating and Extracting Jar files:
$ jar -cvf "filename"(.jar) "src"(.java)  //(cvf - create verbos file)
$ jar -cvf wc WordCount.java

Extract jar:
$ jar -xvf filename.jar 
Change File Permission (chmod):
$ chmod 755 file.txt

7 r w x (root user)
5 r - x (group user)
5 r - x (other user)
There are plenty of commands in linux, but we require only few of them listed above enough to work with hadoop. After practicing above commands, you have enough knowledge of linux command to start working on hadoop.

1 comments:

Post a Comment