Accelerating In Situ Analysis using Non-volatile Memory.

The relatively slower data transfer speeds resulting in I/O bottlenecks in scientific simulations is one of the critical challenges in exascale computing. Simulations generate large data and analysis applications consume this data to provide time-critical insights. The limited size and high power consumption of Dynamic Random Access Memory (DRAM) capacity leaves slow storage devices as the primary option for large-scale data transfers. Non-volatile memory (NVM) devices such as Intel Optane bridges the gap between storage and volatile memory by providing DRAM-comparable performance and persistence. We present PQueue, a data transfer library for in situ analysis of simulation output using persistent memory. PQueue leverages NVM and provides an API that resembles high-level parallel I/O libraries such as PnetCDF to enable seamless transition for application developers. We achieved a maximum of 7 × improvement in write times and a maximum of 10 × improvement in read times as compared to PnetCDF.