Hadoop Shell commands

Hadoop Shell commands:

The File System (FS) shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others.

The FS shell is invoked by:
hadoop fs <args>

Note:
If HDFS is being used, hdfs dfs is a synonym.

List of Hadoop Shell commands:

•     appendToFile
•     cat
•     checksum
•     chgrp
•     chmod
•     chown
•     copyFromLocal
•     copyToLocal
•     count
•     cp
•     createSnapshot
•     deleteSnapshot
•     df
•     du
•     dus
•     expunge
•     find
•     get
•     getfacl
•     getfattr
•     getmerge
•     help
•     ls
•     lsr
•     mkdir
•     moveFromLocal
•     moveToLocal
•     mv
•     put
•     renameSnapshot
•     rm
•     rmdir
•     rmr
•     setfacl
•     setfattr
•     setrep
•     stat
•     tail
•     test
•     text
•     touchz
•     truncate
•     usage
•     Deleting objects
•     Overwriting Objects
•     Timestamps
•     Security model and operations
•     Commands of limited value

1)appendToFile:

Using appendToFile , we can Append single src, or multiple srcs from local file system to the destination file system(HDFS).

Note:
Also can reads input from stdin(console) and appends to destination file system(HDFS).

Syntax:
Usage: hadoop fs -appendToFile <localsrc> ... <dst>


Command: single local source
mano@Mano:~$ hadoop fs -appendToFile /home/mano/DataSets/drug.dat /MANO/Hadoop_shell/appendToFile/patient.dat

Command:Multiple local source
mano@Mano:~$ hadoop fs -appendToFile /home/mano/DataSets/patient.dat /home/mano/DataSets/drug.dat /MANO/Hadoop_shell/appendToFile/patient.dat

Command: Read from stdin input
mano@Mano:~$ hadoop fs -appendToFile - /MANO/Hadoop_shell/appendToFile/patient.dat

using - can read input from stdin

2)cat:
Using cat command to see the source of paths on stdout.

Syntax:
Usage: hadoop fs -cat [-ignoreCrc] URI [URI ...]

-ignoreCrc - to disable checksum verification

Command: single source
mano@Mano:~$ hadoop fs -cat /MANO/Hadoop_shell/appendToFile/pa*

Command: Multiple source
mano@Mano:~$ hadoop fs -cat /MANO/Hadoop_shell/appendToFile/pa* /MANO/Sqoop_import_table/students/par*

3)checksum:
Returns the checksum information of a file.

Syntax:
Usage: hadoop fs -checksum URI

Command:
mano@Mano:~$ hadoop fs -checksum /MANO/Hadoop_shell/appendToFile/pa*

Output:
/MANO/Hadoop_shell/appendToFile/patient.dat    MD5-of-0MD5-of-512CRC32C    00000200000000000000000065f31a0b19e72662cf94a397bd56f2a4

4)chgrp:
using chgrp option , we can the change group level of files to other groups

Syntax:
Usage: hadoop fs -chgrp [-R] GROUP URI [URI ...]

-R option will make the change recursively

Command:
mano@Mano:~$ hadoop fs -chgrp -R manogroup /MANO/Hadoop_shell/appendToFile/pa*

5)chmod:
Change the permissions of files.

Syntax:
Usage: hadoop fs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]

-R option will make the change recursively

Command:symbolic permissions notation
mano@Mano:~$ hadoop fs -chmod -R u=rwx,g=rx,o=r /MANO/Hadoop_shell/appendToFile/pa*

The letters u, g, and o stand for "user", "group", and "other" and letters "r", "w", and "x" stand for "read", "write", and "execute"

Command: Octal representation
mano@Mano:~$ hadoop fs -chmod -R 754 /MANO/Hadoop_shell/appendToFile/pa*

    4 stands for "read",
    2 stands for "write",
    1 stands for "execute", and
    0 stands for "no permission."

6)chown:
Change the owner of files.

Syntax:
Usage: hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]

-R option will make the change recursively

Command:
mano@Mano:~$ hadoop fs -chown -R manohar:manogroup /MANO/Hadoop_shell/appendToFile/pa*