A generic API for load balancing in distributed systems for big data management

Distributed systems for big data management very often face the problem of load imbalance among nodes. To address this issue, there exist almost as many load balancing strategies as there are different systems. When designing a scalable distributed system geared towards handling large amounts of information, it is often not so easy to anticipate which kind of strategy will be the most efficient to maintain adequate performance regarding response time, scalability, and reliability at any time. Based on this observation, we describe a generic API to implement and experiment any strategy independently from the rest of the code, prior to a definitive choice for instance. We then show how existing load balancing strategies used by famous systems could be implemented with this API. We also present how this work has helped us implement load balancing on our distributed system and modify the behavior of our strategy in a few lines of code. This led us to easily perform various experiments to determine the most efficient scheme for our system. This paper is an extension to our work presented at Workshop on Parallel and Distributed Computing for Big Data Applications (WPBA) 2014. We detail here more experiments and extend the use of the API to a broad class of big data storage systems. Copyright (c) 2015 John Wiley & Sons, Ltd.