In the mapper function the process of writing the output doesnot simply involve writing it to the disk. The process involves taking advantage of buffering in memory and presorting in memory for improving efficiency.
Each mapper has a circular memory buffer that it writes the output to. This buffer is defined by "io.sort.mb" and which by default is 100MB.
When the contents of the buffer reaches a certain size defined by the parameter io.sort.spill.percent then the contents of the buffer will be spilled to the disk. Before it writes to the disk a thread divides the data into partitions and performs an in memory sort of keys. If there is a combiner function then it is run on the output of the sort.
Each time the memory buffer reaches the spill threshold, a new spill file is created. So after the map task has completed and its last output has been written, there could be several spill files. The property io.sort.factor says how many of those spill files will be merged into one file at a time.
Before the task is finished the merged into a single partioned and sorted output file.
So if buffer is not filled (may be high io.sort.mb) then the combiner will only run only once at the end of the map task. However if there are multiple spills (affected by io.sort.mb) which is more than io-sort.factor then combiner would be run more than one times.
Friday, June 13, 2014
Monday, May 26, 2014
Hadoop doesnot use Java's Serialization
Serialization involves mapping a runtime object into an ObjectOutputStream, which can then be written to the filesystem or stored in a database. So during object serialization, the default Java serialization mechanism writes the metadata about the object, which includes the class name, field names and types. This class definition is stored as a part of the serialized object. This stored metadata enables the deserialization process to reconstitute the objects and map the stream data into the class attributes with the appropriate type.
Hadoop doesnot use Java's serialization as we are using Writable with Hadoop, so it knows the expected class. Writables don't store their type in the serialized representation as while deserializing, Hadoop know what is expected. eg. if the input key is a LongWritable, so an empty LongWritable instance is asked to populate itself from the input data stream. As no meta info needs to be stored this results in considerably more compact binary files, straightforward random access and higher performance.
Sunday, May 18, 2014
Big Data, Hadoop and HDFS.
Big Data used to describe a massive volume of both structured and unstructured data that is so large that it's difficult to process using traditional database and software techniques.
On a high level it has the following characteristics:
Volume - The large amount of data generated that must be taken into account and analyzed along with its management to make decisions.
Velocity - The fastness in which data is being produced or changed and the speed with which data must be received, understood, and processed.
Variety - It includes structured, unstructured, semi-structured and complex data which creates difficulties in management and governance of these data.
Hadoop is a technology which basically caters to two of the prime problems which big data brings with it: how to store and work with voluminous data and more important how to understand data and turn it into a competitive advantage.
Hadoop essentially fills a gap in this area by effectively storing and providing computational capabilities over substantial amounts of data. It’s a distributed system made up of a distributed file system and it offers a way to parallelize and execute programs on a cluster of machines.
In short Hadoop provides both distributed storage and computational capabilities.
Architecturally, Hadoop is just the combination of two technologies: the Hadoop Distributed File System (HDFS) that provides storage, and the MapReduce programming model which provides processing.
In this context HDFS exists to split, distribute, and manage chunks of the overall data set, which could be a single file or a directory full of files. These chunks of data are pre-loaded onto the worker nodes, which later process them. By having the data local at process time, HDFS saves all of the headache and inefficiency of shuffling data back and forth across the network.
The HDFS has a master-slave architecture where the HDFS master (name node) is responsible for partitioning the storage across the slave nodes (data node) and keeping track of where data is located.
The fundamental principle which governs HDFS are scalability and availability which are achieved by data replication and fault tolerance.
Subscribe to:
Posts (Atom)