Understanding the Memory-Utilization of MPI Libraries: Challenges and Designs in Implementing the MPI_T Interface

The MPI Tools information interface (MPI_T), introduced as part of MPI 3.0 standard, has been gaining momentum in both the MPI and performance tools communities. In this paper, we investigate the challenges involved in profiling the memory utilization characteristics of MPI libraries that can be exposed to tools and libraries leveraging the MPI_T interface. We propose three design alternatives to enable such profiling from within MPI, and study their viability in light of these challenges. We analyze the benefits and shortcomings of each of them in detail, with a particular focus on the performance and memory overheads that they introduce. We evaluate the performance and scalability of these designs using micro-benchmarks, MPI-level benchmarks and applications. The overheads of the proposed design amounts to just 0.8% of the MILC application runtime with 4,096 processes. The paper also presents a case study that uses the MPI_T memory profiling information in MVAPICH2 to optimize the memory utilization of UH3D application runs, where memory savings of up to 7.3x was achieved.