Map-Balance-Reduce: An improved parallel programming model for load balancing of MapReduce

With the advent of the era of big data, the demand of massive data processing applications is also growing. Currently, MapReduce is the most commonly used data processing programming model. However, in some data processing cases, it has some defects. MapReduce programming based on key/value pairs, matches the output of the Map tasks that will be transported to Reduce task nodes. The data with same key can only be processed by a Reduce node. If the data corresponding to a particular key or several keys accounts for most of all data, the Reduce node task will generate unbalanced load. In view of this defect, this paper proposes a new parallel programming model—Map-Balance-Reduce (MBR) programming model. It runs on our improved Hadoop framework and can effectively process the special data with unbalanced keys. This paper is based on two different scheduling, the processing and self-adaption scheduling. These two scheduling are designed to achieve MBR programming model. The actual testing results show that compared with MapReduce programming model, the MBR programming model under Hadoop can achieve the improvement of 9.7% to 17.6% in efficiency when testing data distributes unevenly. Furthermore, when testing conventional even-distributed data, it will only bring 1.02% time cost.