Performance Characterization Of Network Mechanisms For Non-Contiguous Data Transfers In Mpi

Message Passing Interface (MPI) is a very popular parallel programming model for developing parallel scientific applications. The complexity of data handled by scientific applications often results in their placement in non-contiguous locations in memory. In order to handle such complex and non-contiguous data, domain scientists often use user-defined datatypes that are supported by the MPI standard through Derived Data Types (DDT). Traditionally, popular implementations of the MPI standard have used simple schemes to "pack" and "unpack" non-contiguous data to and from contiguous memory regions before and after communication operations. On the other hand, vendors of high-performance interconnects have introduced several hardware offloaded schemes to perform optimized transfers of non-contiguous data. Although researchers have attempted to characterize the performance of non-contiguous transfers in the past, they have not gone in depth into the communication runtime to see where the bottlenecks lie, especially in the presence of network offloaded support. In this paper, we take up this challenge and evaluate different designs for non-contiguous data transfers on a particular MPI run-time using our synthetic benchmarks. We consider the following designs: 1) Pack-Unpack based RDMA transfer 2) User mode Memory Registration (UMR) based RDMA transfer. 3) Pipelined transfer. 4) SGL based transfer. For each of these designs, we measure the impact of a) serialization, b) memory registration, c) Packing, and d) additional overheads to send. From these evaluations, we realize why MPI run-times may not meet the expectations of DDT, and when to use DDT based implementations.