Analysis of MPI Communication Time for Distribution of Repartitioned Data.

Repartitioning in a parallel setting can be defined as the task of redistributing data across processes based on a newly imposed grid/layout. Repartitioning is a fundamental problem, with applications in domains that typically involve computation on tiles (blocks/patches) of varying resolution, for example, while creating multiresolution data formats in in-situ mode (such as the JPEG format and its variants). This paper explores the performance and tradeoffs of different ways to perform the data redistribution phase. We explore a greedy scheme that aims to minimize data movement while compromising on load balancing and a balanced scheme that aims to create a balanced load across processes while compromising on data movement. For both schemes, we further compare per-patch (staggered data transfer) and per-rank (aggregated data transfer) communication patterns to measure the impact of buffer size on MPI point-to-point communication performance. We conclude that the reduced data movement of the greedy scheme leads to reduced transfer times during redistribution. We further conclude that the per-patch communication pattern outperforms per-rank communication.