Scalable RMA-based Communication Library Featuring Node-local NVMs

Remote Memory Access (RMA) is a useful communication interface to develop high-performance applications with complicated communication patterns. However, the data scales of such applications are still limited by the totally available main memory capacity. To accommodate extreme scale executions of those applications, we developed vGASNet, which is an RMA-based communication library that exploits the capacity of non-volatile memory (NVM) on each node. With vGASNet, NVM devices on nodes compose a large shared address space. Under this model, the key for good application performance is to reduce bandwidth bottlenecks. First, since NVM is much slower than DRAM, reducing the amounts of NVM accesses is important. For this purpose, vGASNet regards DRAM of each computation node as a cache of NVM. Next, one of bottleneck sources in RMA is caused by access contention. In order to mitigate its effects, vGASNet adopts cooperative cache mechanism, which make multiple caches of an object on several nodes. Our evaluation using vGASNet shows the above cache mechanism improves the scalability of RMA.