Monday, May 26, 2014

Hadoop doesnot use Java's Serialization

Serialization involves mapping a runtime object into an ObjectOutputStream, which can then be written to the filesystem or stored in a database. So during object serialization, the default Java serialization mechanism writes the metadata about the object, which includes the class name, field names and types. This class definition is stored as a part of the serialized object. This stored metadata enables the deserialization process to reconstitute the objects and map the stream data into the class attributes with the appropriate type.

Hadoop doesnot use Java's serialization as we are using Writable with Hadoop, so it knows the expected class. Writables don't store their type in the serialized representation as while deserializing, Hadoop know what is expected. eg. if the input key is a LongWritable, so an empty LongWritable instance is asked to populate itself from the input data stream. As no meta info needs to be stored this results in considerably more compact binary files, straightforward random access and higher performance.

Sunday, May 18, 2014

Big Data, Hadoop and HDFS.

Big Data used to describe a massive volume of both structured and unstructured data that is so large that it's difficult to process using traditional database and software techniques.
On a high level it has the following characteristics:
Volume - The large amount of data generated that must be taken into account and analyzed along with its management to make decisions.
Velocity - The fastness in which data is being produced or changed and the speed with which data must be received, understood, and processed.
Variety - It includes structured, unstructured, semi-structured and complex data which creates difficulties in management and governance of these data.

Hadoop is a technology which basically caters to two of the prime problems which big data brings with it: how to store and work with voluminous data and more important how to understand data and turn it into a competitive advantage.
Hadoop essentially fills a gap in this area by effectively storing and providing computational capabilities over substantial amounts of data. It’s a distributed system made up of a distributed file system and it offers a way to parallelize and execute programs on a cluster of machines.
In short Hadoop provides both distributed storage and computational capabilities.

Architecturally, Hadoop is just the combination of two technologies: the Hadoop Distributed File System (HDFS) that provides storage, and the MapReduce programming model which provides processing.
In this context HDFS exists to split, distribute, and manage chunks of the overall data set, which could be a single file or a directory full of files. These chunks of data are pre-loaded onto the worker nodes, which later process them. By having the data local at process time, HDFS saves all of the headache and inefficiency of shuffling data back and forth across the network.
The HDFS has a master-slave architecture where the HDFS master (name node) is responsible for partitioning the storage across the slave nodes (data node) and keeping track of where data is located.
The fundamental principle which governs HDFS are scalability and availability which are achieved by data replication and fault tolerance.