Continual and Cost-Effective Partitioning of Dynamic Graphs for Optimizing Big Graph Processing Systems

Recently, several cluster computing frameworks have been proposed for scalable and efficient processing of big graphs. The manner in which graph data is partitioned and placed on the compute nodes has a significant impact on cluster performance. While most existing graph partitioning and placement strategies have been designed for static graphs, the graphs in many modern applications are dynamic (time-evolving). In this paper, we propose a unique, continuous and multi-cost sensitive approach for partitioning dynamic graphs. Our approach incorporates novel cost functions that take into account major factors that impact the performance of big graph processing clusters. We also present incremental algorithms to efficaciously handle various types of graph dynamics. Our algorithms are unique in that they work by locally adjusting the partitions thus avoiding massive repartitioning. This paper reports a series of experiments to demonstrate the effectiveness of the proposed algorithms in maximizing the performance of big graph processing systems on dynamic graphs.