Monday, May 26, 2014

Hadoop doesnot use Java's Serialization

Serialization involves mapping a runtime object into an ObjectOutputStream, which can then be written to the filesystem or stored in a database. So during object serialization, the default Java serialization mechanism writes the metadata about the object, which includes the class name, field names and types. This class definition is stored as a part of the serialized object. This stored metadata enables the deserialization process to reconstitute the objects and map the stream data into the class attributes with the appropriate type.

Hadoop doesnot use Java's serialization as we are using Writable with Hadoop, so it knows the expected class. Writables don't store their type in the serialized representation as while deserializing, Hadoop know what is expected. eg. if the input key is a LongWritable, so an empty LongWritable instance is asked to populate itself from the input data stream. As no meta info needs to be stored this results in considerably more compact binary files, straightforward random access and higher performance.

No comments:

Post a Comment