About Big Data

Before to start exploration on Big data, let's think about how the big data comes, in picture??

The below are the reasons behind the big data comes in picture:

1. Evolution of technology
2. IOT(Internet Of Things)
3. Social Media
4. Other factors

Let's see briefly:
 
1)Evolution of technology:

Earlier we had land line phones, But nowadays,we have android,IOS smart phones, to make our life smarter. so just think, for each  operation which we perform on smart phones, generates a data, that resides somewhere

Desktops are the source to handle operations, i mean to store and process using storage devices like floppy,discs,taps,..etc.,

But in these days, Hard disks,cloud storage plays a vital role.

Earlier , we are in the hand of Analog storage, but these days almost of Digital storage. and also about the evolution of car, self driving car,

2)IOT(Internet Of Things):
IOT connects physical device to Internet and makes device smarter.

Example:
Smart TV's, Smart Ac's, Smart Car's etc.,

3)Social Media:
Data generation on social media sites,

• Facebook likes,videos,photos,tags,comments etc.,
• Tweeter tweets,
• Youtube video uploads
• Instagram pics,
• Emails

4)Other Factors:

• Retail
• Banking & Finance,
• Media & Entertainment
• Health care,
• Education areas,
• Government,
• Transportation, Insurance etc.,

Note: Assumption by 2020, 50 billions IOT devices will in the world.

Big Data:

Big data is a term for data sets that are so large or complex or even huge or massive volume of both structured and unstructured data that traditional data processing application software is inadequate to deal with big data or difficult to process.

Note: Big Data is not a technology, it's paradigm(pattern) shift.


To determine which data is considered as Big Data,  we have some
Characteristics of big data:(5V's of Big Data):

1)V- Volume:
Amount of data being generating and generated.

2)V- Variety:
Different kinds of data , that is being generated from various sources.

Types of data:

1. Structured data - Tables
2. Semi-structured data - CSV,JSON,EMAILS,TSV,XML
3. Unstructured data - Videos, images, Logs, Audio files

3)V- Velocity:
The speed at which the data is being generated and processed to meet the demands.
Data is being generated at alarming rate.

4)V- Value:
Mechanism to bring the correct meaning out of huge data.

5)V- Veracity:
Uncertainty and inconsistencies in the data, i.e., The quality of captured data can vary greatly, affecting accurate analysis.

Problems with Big Data:

Problem 1:Storing exponentially growing large data sets in a non-distributed system.

Problem 2:Processing variety of data i.e., complex structure data.

Problem 3:Processing data faster

To put a solution for those above problems , Hadoop comes and plays a vital role.

Solutions with Hadoop:

Problem 1:Storing exponentially growing large data sets in a non-distributed system.

Solution: HDFS

• It is storage part of Hadoop
• Distributed File system,
• Divides files into smaller chunks and stores across the cluster.
• Scalable as per requirement(Scalability)

Problem 2:Storing varies of data.

Solution: HDFS

• HDFS allows to store any kind of data,(Structured,semi-structured or unstructured)
• No schema validation in HDFS while dumping data
• Follows WORM (Write once Ream Many)

Problem 3:Processing data faster

Solution: MapReduce

• Parallel execution of data present in HDFS
• Allows to process the data locally, i.e., each node responsible for data processing which stored on it.

Big data as an opportunity to bring below:

Big data use cases:
Below are some of the Big data use cases from different domains:

•  Improve Customer Experience
•  Sentiment analysis
•  Customer Churn analysis
•  Predictive analysis
•  Real-time ad matching and serving

                                                                                                                                                  Next_Page