Accelerating I/O performance of ZFS-based Lustre file system in HPC environment

To meet increasing data access performance demands of applications run on high-performance computing (HPC) systems, an efficient design of HPC storage file system is becoming more important. Nowadays, Lustre is the most commonly used distributed parallel file system that has been deployed on world’s fastest supercomputers and currently supports two backend file systems: ldiskfs and ZFS. Although ZFS provides numerous performance optimization options, most of the supercomputer systems use Lustre with ldiskfs for higher performance. In this work, we analyze the root cause of low I/O performance on a ZFS-based Lustre file system and propose a novel ZFS scheme, dynamic-ZFS, which combines two optimization approaches. The experiment results show that our approach can improve the sequential I/O performance by 37% on average. We demonstrate that dynamic-ZFS can deliver an I/O performance comparable to that of ldiskfs-based Lustre while still providing a multitude of beneficial features.