A Zero-Penalty Container-Based Execution Infrastructure For Hadoop Framework

With the growing need of sharing computing resoures in data-center, various resource management and scheduling schemes have been designed for Hadoop framework. Unfortunately, the Hadoop's existing schedulers (such as Fair and Delayed Fair) suffer from the conflicting nature of fairness and data locality, which can jeopardize system performance. In this work, we propose a zero-penalty container-based execution infrastructure for Hadoop to address such a problem. Specifically, based on Linux container LXC, we designed the framework where map/reduce tasks can be executed inside their designated containers instead of in virtual machines as in the traditional approach. When a task needs to be cancelled due to system fairness requirements, unlike the conventional approach in Hadoop that kills the task, the corresponding container is frozen, which can continue its execution when CPU slots become available for the task at a later time. With such un/freeze operations of LXC containers, the proposed infrastructure can essentially provide preemptive executions of map/reduce tasks in Hadoop framework, where the partial computation of the enclosed tasks in containers can be preserved with zero-penalty. We illustrated the capabilities of the proposed execution infrastructure through a simple setting of two users, and the results show that the proposed scheme can remarkably reduce the waiting and completion time of tasks by up to 32.89% and 14.93%.