Hadoop Illuminated

What is Big Data?

You probably heard the term Big Data — it is one of the most hyped terms now. But what exactly is big data?

Big Data is very large, loosely structured data set that defies traditional storage

Human Generated Data and Machine Generated Data

Human Generated Data is emails, documents, photos and tweets. We are generating this data faster than ever. Just imagine the number of videos uploaded to You Tube and tweets swirling around. This data can be Big Data too.

Machine Generated Data is a new breed of data. This category consists of sensor data, and logs generated by ‘machines’ such as email logs, click stream logs, etc. Machine generated data is orders of magnitude larger than Human Generated Data.

Before ‘Hadoop‘ was in the scene, the machine generated data was mostly ignored and not captured. It is because dealing with the volume was NOT possible, or NOT cost effective.

Where does Big Data come from

Original big data was the web data — as in the entire Internet! Remember Hadoop was built to index the web. These days Big data comes from multiple sources.

• Web Data — still it is big data
• Social media data : Sites like Facebook, Twitter, LinkedIn generate a large amount of data
• Click stream data : when users navigate a website, the clicks are logged for further analysis (like navigation patterns). Click stream data is important in on line advertising and and E-Commerce
• sensor data : sensors embedded in roads to monitor traffic and misc. other applications generate a large volume of data
• Connected Devices : Smart phones are a great example. For example when you use a navigation application like Google Maps or Waze, your phone sends pings back reporting its location and speed (this information is used for calculating traffic hotspots). Just imagine hundres of millions (or even billions) of devices consuming data and generating data.