LS-HTC: an HTC system for large-scale jobs

High throughput computing (HTC) uses mass computing resources over long periods of time to accomplish a batch of short fast jobs, it is widely employed by Simulation Computation such as Earth Science, Materials Science, Biomedical Science to process large scale simulation tasks. When the number of jobs reaches a large-scale level, such as millions or tens of millions, the scheduling and management of massive tasks will bring great burden to the high performance computing (HPC) cluster. Therefore, an HTC system that supports large-scale jobs with few impact on HPC cluster becomes an urgent need for these communities. To address this problem, we propose an LS-HTC system which can schedule million-level jobs and million-level computing resources. The architecture and workflow of LS-HTC is designed, and a two-level scheduling solution is provided for large-scale jobs execution. Prototype system is achieved then evaluated using more than 20 million jobs and 8000 compute nodes and 128,000 CPU cores at our HPC cluster. Experimental results indicate that the LS-HTC system can take best usage of computing resources by dynamically adjusting the sum of compute nodes according to the sum of jobs with negligible influence on shared storage system and management system of HPC cluster.