HDFS Write and Read Mechanism:

HDFS Write Mechanism:

HDFS follow the below two layers

1. Pipeline setup
2. Acknowledgment

 

HDFS Read Mechanism:

Previous Page:                                                                 

Differences between the ways to start/stop Hadoop daemon processes

Differences between the ways to start/stop hadoop daemon processes:

Three ways to run the hadoop daemon process

1. start-all.sh and stop-all.sh
2. start-dfs.sh,stop-dfs.sh and start-yarn.sh,stop-yarn.sh
3. hadoop-daemon.sh start/stop namenode/datanode and yarn-deamon.sh start/stop resourcemanager

1.start-all.sh and stop-all.sh : 

start-all.sh & stop-all.sh ==> used to start and stop hadoop daemons all at once. 

Note:
Running on the master machine will start/stop all daemons on all the nodes of a cluster.

2.start-dfs.sh, stop-dfs.sh and start-yarn.sh, stop-yarn.sh : 

It is also start/stops all the the daemons, but HDFS and YARN daemons separately on all the nodes from the master machine.

3. hadoop-daemon.sh start namenode/datanode and yarn-deamon.sh start/stop resourcemanager :

To start individual daemons on an individual machine manually. You need to go to a particular node and run these commands.

Use case:

Suppose you have added a new node to your cluster and need to start the DataNode daemon only on this machine, 

sbin/hadoop-daemon.sh start datanode

Hadoop Shell commands

Hadoop Shell commands:

The File System (FS) shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others.

The FS shell is invoked by:
hadoop fs <args>

Note:
If HDFS is being used, hdfs dfs is a synonym.

List of Hadoop Shell commands:

•     appendToFile
•     cat
•     checksum
•     chgrp
•     chmod
•     chown
•     copyFromLocal
•     copyToLocal
•     count
•     cp
•     createSnapshot
•     deleteSnapshot
•     df
•     du
•     dus
•     expunge
•     find
•     get
•     getfacl
•     getfattr
•     getmerge
•     help
•     ls
•     lsr
•     mkdir
•     moveFromLocal
•     moveToLocal
•     mv
•     put
•     renameSnapshot
•     rm
•     rmdir
•     rmr
•     setfacl
•     setfattr
•     setrep
•     stat
•     tail
•     test
•     text
•     touchz
•     truncate
•     usage
•     Deleting objects
•     Overwriting Objects
•     Timestamps
•     Security model and operations
•     Commands of limited value

1)appendToFile:

Using appendToFile , we can Append single src, or multiple srcs from local file system to the destination file system(HDFS).

Note:
Also can reads input from stdin(console) and appends to destination file system(HDFS).

Syntax:
Usage: hadoop fs -appendToFile <localsrc> ... <dst>


Command: single local source
mano@Mano:~$ hadoop fs -appendToFile /home/mano/DataSets/drug.dat /MANO/Hadoop_shell/appendToFile/patient.dat

Command:Multiple local source
mano@Mano:~$ hadoop fs -appendToFile /home/mano/DataSets/patient.dat /home/mano/DataSets/drug.dat /MANO/Hadoop_shell/appendToFile/patient.dat

Command: Read from stdin input
mano@Mano:~$ hadoop fs -appendToFile - /MANO/Hadoop_shell/appendToFile/patient.dat

using - can read input from stdin

2)cat:
Using cat command to see the source of paths on stdout.

Syntax:
Usage: hadoop fs -cat [-ignoreCrc] URI [URI ...]

-ignoreCrc - to disable checksum verification

Command: single source
mano@Mano:~$ hadoop fs -cat /MANO/Hadoop_shell/appendToFile/pa*

Command: Multiple source
mano@Mano:~$ hadoop fs -cat /MANO/Hadoop_shell/appendToFile/pa* /MANO/Sqoop_import_table/students/par*

3)checksum:
Returns the checksum information of a file.

Syntax:
Usage: hadoop fs -checksum URI

Command:
mano@Mano:~$ hadoop fs -checksum /MANO/Hadoop_shell/appendToFile/pa*

Output:
/MANO/Hadoop_shell/appendToFile/patient.dat    MD5-of-0MD5-of-512CRC32C    00000200000000000000000065f31a0b19e72662cf94a397bd56f2a4

4)chgrp:
using chgrp option , we can the change group level of files to other groups

Syntax:
Usage: hadoop fs -chgrp [-R] GROUP URI [URI ...]

-R option will make the change recursively

Command:
mano@Mano:~$ hadoop fs -chgrp -R manogroup /MANO/Hadoop_shell/appendToFile/pa*

5)chmod:
Change the permissions of files.

Syntax:
Usage: hadoop fs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]

-R option will make the change recursively

Command:symbolic permissions notation
mano@Mano:~$ hadoop fs -chmod -R u=rwx,g=rx,o=r /MANO/Hadoop_shell/appendToFile/pa*

The letters u, g, and o stand for "user", "group", and "other" and letters "r", "w", and "x" stand for "read", "write", and "execute"

Command: Octal representation
mano@Mano:~$ hadoop fs -chmod -R 754 /MANO/Hadoop_shell/appendToFile/pa*

    4 stands for "read",
    2 stands for "write",
    1 stands for "execute", and
    0 stands for "no permission."

6)chown:
Change the owner of files.

Syntax:
Usage: hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]

-R option will make the change recursively

Command:
mano@Mano:~$ hadoop fs -chown -R manohar:manogroup /MANO/Hadoop_shell/appendToFile/pa*

Hadoop

Apache Hadoop: 

Hadoop is a Open source software framework which allows us to store and process large data sets in parallel and distributed fashion.

Note:
The library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

We might wonder, Why Hadoop for BigData??
Below are the reasons of it,

• Distributed Computing
• Parallel processing
• Low latency
• Data availability
• Fault tolerance
• Cross platform

There are two parts of Hadoop:

1. Hadoop Distributed File System(HDFS)
2. MapRedue

1)Hadoop Distributed File System(HDFS):
The Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications.

Three main HDFS concepts are,
1. Sharding
2. Distributed
3. Replicated Factor

1.Sharding :

Sharding means single file splited into multiple blocks.

Highlights:

In Hadoop version 1, Default block size is 64 Mb
In Hadoop version 2, Default block size is 128 Mb
But the block size is not a fied size, we can increase or decrease the block size.

Note:
the block size  can be set and change in hdfs-site.xml file

2.Distributed:
Each block of block from the sharding, it stores across the cluster on different machines

3.Replicated Factor:

Each block are replicated(copies) among the cluster.

• Default replication is 3.
• Minimum number of replication is 1.
• Maximun number of relication is 512.

Note:
<dfs.replication> - here we can change the replication factor.

These are the reason with usage of HDFS concepts:

• Data Availability

• Fault Tolerance

- its work based on replication factor

• Parallel Processing

- it work based on distribution.

Note:
HDFS follows the MASTER/SLAVE architecture,

HDFS Core components:(Daemons)

• NameNode
• DataNode
• Secondary NameNode

i)NameNode:

• It is a Master Node.

• Maintains and manages all the datanodes.

• It stores only Metadata not  an actual data. i.e., information about data blocks, Ex: location of blocks, size of file,permissios etc.,

• Receives heartbeat and block reports every 3 senconds

• By default, only one namenode. so it is a single point of failure, CHA-High availabilty (HA)--> Active -->standby

NameNode stores metadata in two files:

1. FsImage  - stores all changes and modifications in HARDDISK
2. editsLog - recent changes info, in RAM

ii)DataNode:

• It is Slave Node.
• Stores actual data.
• Serves read and write from the clients.

Note:

Version 1 – 4000 datanoes.
Version 2 - 35000 datanodes.

Highlight:

If the heart beat signal is not received in 3sec, then the namenode will wait for 10 mins. If the signal is not recived yet 10 mins then it declare the datanode is deadnode.

iii)Secondary NameNode:

• It is the checkpoint node, to combine the ediLog and FsImage file to form persistenced FsImage,
• it does periodically (defualt 1 hr)
• to make NameNode to be avaible with recent changes

2)MapRedue:

MapReduce is a programming framework that allows us to perform distributed and parallel processing of large data sets in distributed environment.

Note:

• Process like Divide & Conquer method (Divide the job & combine the result). 
• Input / Output for MapReduce --> HDFS ( Read the data from HDFS & Write the output to HDFS ).

Version 1 - mapreduce daemons:

• JobTracker
• TaskTracker

iv)JobTracker:

• Co-ordinate,
• Failure handling,
• Resource management,
• Scheduling.

v)TaskTracker: 

It perform a task, which is assigned by the jobtracker.

Version 2 - Yarn daemons (yet another resource negotiator):

• ResourceManger
• NodeManager

iv)ResourceManger:

• Role of jobtracker. 

v)NodeManager:

• Role of tasktracker.

Why we use MapReduce:

• To process the unstructed data to structured data for analysis,
• Large datasets processing can be done in parallel and distributed

Four task in MapReduce:

1. Map Task
2. Reducer Task
3. Combiner Task
4. Partitioner Task

1.Map Task:

• Assigns the task to the worker.
• Number of mapper = Number of blocks/ Input split.

2. Reducer Task:

• To perform Aggragate Functions.
• Number of reducer = Number of output files.
• Select e_name, e_id from employee  --> Map only Job.
• Select e_name, e_id, sum (e_salary) from employee --> Map & Reduce Job.

3. Combiner Task:

•  Local reducer inside Mapper Class

4.  Partitioner Task:

Data split to increase the performance

Two types of partitions:
Default Partition: It follows based on Hash value
Custom Partition:We decide on which data to be partitioned.

Note:
Both for Optimize

Previous Page                                                                                   Next Page:

Hadoop Common Commands:

Hadoop Common Commands:

Hadoop help usage:

command:
mano@Mano:~$ hadoop -help

1)archive:
Hadoop Archives:

Archives are to the problem with small files in Hadoop:

Hadoop works best with big files and small files are handled inefficiently in HDFS.  As we know, Namenode holds the metadata information in memory for all the files stored in HDFS. Let’s say we have a file in HDFS which is 1 GB in size and the Namenode will store metadata information of the file – like file name, creator, created time stamp, blocks, permissions etc.

Now assume we decide to split this 1 GB file in to 1000 pieces and store all 1000 “small” files in HDFS. Now Namenode has to store metadata information of 1000 small files in memory. This is not very efficient – first it takes up a lot of memory and second soon Namenode will become a bottleneck as it is trying to manage a lot of data.

What is HAR?

Hadoop Archives or HAR is an archiving facility that packs files in to HDFS blocks efficiently and hence HAR can be used to tackle the small files problem in Hadoop. A Hadoop archive always has a *.har extension.

Note:
HAR is created from a collection of files and the archiving tool (a simple command) will run a MapReduce job to process the input files in parallel and create an archive file.

How to create HAR file:

Syntax:
Usage: hadoop archive -archiveName name -p <parent> [-r <replication factor>] <src>* <dest>

Command:
mano@Mano:~$ hadoop archive -archiveName Mano.har -p /MANO/Sqoop_import_table/employees /HAR

How to list HAR file :
Command:
mano@Mano:~$ hadoop fs -ls /HAR/Mano.har
Found 4 items
-rw-r--r--   1 mano supergroup          0 2017-09-02 16:44 /HAR/Mano.har/_SUCCESS
-rw-r--r--   5 mano supergroup        446 2017-09-02 16:44 /HAR/Mano.har/_index
-rw-r--r--   5 mano supergroup         22 2017-09-02 16:44 /HAR/Mano.har/_masterindex
-rw-r--r--   1 mano supergroup         65 2017-09-02 16:43 /HAR/Mano.har/part-0

 A Hadoop archive directory contains metadata (in the form of _index and _masterindex) and data (part-*) files. The _index file contains the name of the files that are part of the archive and the location within the part files.

2)checknative
This command checks the availability of the Hadoop native code

Syntax:
Usage: hadoop checknative [-a] [-h]

-a- To Check all libraries are available.
-h - prints help  message
commands:
mano@Mano:~$ hadoop checknative -a

mano@Mano:~$ hadoop checknative -h

3)classpath
prints the class path needed to get the Hadoop jar and the required libraries

Syntax:
Usage: hadoop classpath [--glob |--jar <path> |-h |--help]


--glob argument helps - to view the class path without wildcards.

Command:
mano@Mano:~$ hadoop classpath

mano@Mano:~$ hadoop classpath --glob

4)distch
Change the ownership and permissions on many files at once.

Syntax:
Usage: hadoop distch [-f urilist_url] [-i] [-log logdir] path:owner:group:permissions

COMMAND_OPTION	Description
-f	List of objects to change
-i	Ignore failures
-log	Directory to log output

5)distcp - distributed copy
Distributed copy file or directories recursively.

distcp is a general utility for copying large data sets between distributed file systems within and across clusters. The distcp command submits a regular MapReduce job that performs a file-by-file copy.

Example:

Copy a directory from one node in the cluster to another
hadoop fs -distcp hdfs://namenodeA/apache_hadoop hdfs://namenodeB/hadoop

6)fs
It is a synonym for hdfs dfs when HDFS is in use.

7)jar
Runs a jar file.

Syntax:
Usage: hadoop jar <jar> [mainClass] args...

8)version

Prints the hadoop version

Command:
mano@Mano:~$ hadoop version

9)CLASSNAME
Runs the class named CLASSNAME. The class must be part of a package.

Syntax:
Usage: hadoop CLASSNAME

10)dfsadmin -safemode

Basically safemode makes only the Read access

Commands:
1)To get help/to see available options
hadoop@Mano:~$ hadoop dfsadmin -safemode

2) To enter into safemode
hadoop@Mano:~$ hadoop dfsadmin -safemode enter

3)To leave safemode
hadoop@Mano:~$ hadoop dfsadmin -safemode leave

4)To list safemode
hadoop@Mano:~$ hadoop dfsadmin -safemode get

5)To wait and list safemode
hadoop@Mano:~$ hadoop dfsadmin -safemode wait

Previous Page:                                                                                                                        Next Page