Efficient Resource Allocation using a Multi-criteria approach and nodes Clustering for Heterogeneous Hadoop Cluster

Hadoop is an open-source framework that is widely used to store and process large amounts of data. Its core component is called YARN that is responsible for resource management and job scheduling for Big Data applications. Although YARN has very advanced processing performance. It suffers from resource allocation problems when used in a heterogeneous cluster serving multiple users. To deal with this heterogeneity and overcome the degradation of cluster, load balancing has become a major challenge for Hadoop run-time management. In this paper, an efficient resource allocation strategy combining multi-criteria decision-making and accurate clustering of nodes in a heterogeneous Hadoop environment is proposed. A multi-criteria decision approach relying on a modified analytical hierarchy process method is applied to assign a score to jobs depending on the resources requirements at run time. First, the proposed system profiles the available nodes by grouping them into clusters with similar performance. Second, the system performs a dynamic multi-criteria selection on resource requests by assigning scores to jobs according to their resource usage. Third, node groups and job scores are used to dynamically allocate jobs to the most appropriate resources in real time while keeping load balance in the heterogeneous cluster. The aims of this present paper are to optimize the resource utilization and achieve a high performance of the Yarn resource management by considering a heterogeneous environment. The experimental results demonstrate that the proposed algorithm provides the best utilization of available resources by 45% and 12% compared to fair and H-fair respectively, which implies a minimum job execution time in a heterogeneous cluster.