A scalable MPI implementation of an ad-hoc parallel I / O system

This paper presents the novel design, implementation and ev aluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS i s the first scalable parallel I/O system completely implement ed in Message Passing Interface (MPI). The MPI implementati on brings the advantages of portability, scalability and high-perfo rmance. AHPIOS allows MPI applications to dynamically mana ge nd scale distributed partitions in a convenient way. The configurati on of both MPI-IO and storage management system is unified and allows for a tight integration of the optimizations of these layers . AHPIOS partitions are elastic: they conveniently scale up and down with the number of resources. We develop two collective I/O s trategies, which leverage a two-tiered cooperative cache i n order to exploit the spatial locality of data-intensive parallel ap plications. The file access latency is hidden from the applic ations through an asynchronous data staging strategy. The two-tiered cooper ativ cache scales with both number of processors and storag e resources. Our experimental section demonstrates that with various op timizations integrated AHPIOS offers a substantial perfor mance benefit over the traditional MPI-IO solutions on both PVFS or Lustre parallel file systems. Index Terms Parallel I/O, Parallel Systems, Distributed File Systems, Parallelism and Concurrency. I. I NTRODUCTION The ever increasing gap between the performance of processo rs and persistent storage devices has focused the attention of several research projects on providing scalable, high-performanc e storage solutions. Additionally, the scale of modern supe rcomputers has recently augmented to performance surpassing the PetaFLOP milestone. A recent survey of High-Performance Computing ( HPC) open science applications [7] has shown that the vast majority of applications are implemented in Message Passing Interface (MPI), which has become the de-facto standard of scalable parallel progr amming. Despite increasing interest in alternative parall el programming paradigms, there are at least two factors that make MPI indis pen able: the limited scalability of shared memory machine s (hence the need for the message passing paradigm on distributed mem ory achines), and the legacy of a huge amount of robust scien tific libraries developed in MPI. A large subset of parallel scientific applications is data in ensive. Parallel file systems represent traditional scala ble high-performance solutions to the storage bottleneck problem. A typical para llel file system stripes files across several independent I/O servers in order to allow parallel file access from many compute nodes simulta neously. Examples of popular file systems include GPFS [43], PVFS [34] and Lustre [20]. These parallel file systems manage the s torage of the vast majority of clusters and supercomputers f rom the top 500 list [19]. They are also used by small and medium cluster o f computers, typically for parallel data processing and vis ualizations. These applications could benefit from a simple transparent s olu ion offering scalable parallel I/O with practically no c st for installing and maintaining a parallel file system. Many parallel applications access the final storage through parallel I/O libraries including MPI-IO [37], Hierarchica l Data Format (HDF) [9], and parallel NetCDF [33]. Both HDF and parallel Ne tCDF are implemented on top of MPI-IO, as MPI-IO handles the low-level file access to the file system. Being part of MPI-2 st andard, MPI-IO defines a set of application programming inte rfac s (API) for file access in parallel. It also specifies the rules f or data consistency. Recently, there is a growing interest in High Productivity C omputer Systems [17]. Besides the performance, the product ivity is taken into consideration, which includes the costs of programmin g, debugging, testing, optimization and administration. I n case of parallel I/O, it gets more difficult to obtain optimal performance fro m the underlying parallel file system, given different I/O re quirements. The default file system configuration cannot always provide an op timal throughput for different data intensive application s. File system reconfiguration may be a solution, but an expensive one, that would inevitably involve the administrative overhead, dat a relocation, and system down time. Additionally, the design complexity o f a distributed parallel file system such as GPFS, makes diffic ult to address the requirements from different I/O patterns at file system level. In other words, implementing the solutions fo r these problems in the file systems would come at the cost of additional comple xity. However, the solutions addressing specific I/O requir ements can be done at a higher level, closer to the applications. This paper presents AHPIOS, a light-weight ad-hoc parallel I/O system that addresses some of the issues mentioned above . AHPIOS can be used as a middleware located between MPI-IO and distri buted storage resources, providing high-performance scal able access to files. AHPIOS can be used as a light-weight low-cost altern ative to any parallel file system. The main goals of AHPIOS des ign are the following: High-performance. High performance is achieved through the tight integration of MPI-IO and storage system, which allows an efficient data access through a two-tiered cooperative cach e nd an asynchronous data staging strategy.