Apala: Adaptive Partitioning and Load Balancing for State-Transition Applications

This paper proposes a partitioning and load-balance scheme for parallelizing state-transition applications on computer clusters. Existing schemes insufficiently balance both the computation of complex state-transition algorithms and the increasing volume of scientific data simultaneously. Apala addresses this problem by introducing the time metric to unify the workloads of computation and data. System profiles in terms of CPU and I/O speeds are considered for accurate workload estimations. Apala consists of two major components: (1) an adaptive decomposition scheme that uses the quad-tree structure to break up workloads and manage data dependencies, (2) a decentralized scheme for distributing workloads across processors. Experimental results from the real-world weather data demonstrate that Apala outperforms other partitioning schemes, and can be readily ported to diverse systems with satisfactory performance.