Architecture of an On-Time Data Transfer Framework in Cooperation with Scheduler System.

Technological advancement in networking and IoT have given researchers new methods or techniques to perform numerical analysis and simulation with the latest data observed on IoT sensors and other measurement devices. In general, large-scale simulations necessitate high-performance computing (HPC) systems. Such HPC systems are operated in a shared manner among researchers. Therefore, it becomes inherently difficult for researchers to use the latest observation data generated on remote data sources for their simulations. To enable researchers to utilize fresh data on a remote data source for computation, we propose an on-time data transfer framework that enables the execution of jobs with fresh data generated on a remote site data on a shared HPC system by extending the SLURM scheduler. The proposed framework consists of two functions: Job pinning and On-time data transfer . With the job pinning function, the proposed framework prevents the scheduling algorithm from rearranging the scheduled start time of jobs. The on-time data transfer function is in charge of data transfer from a remote site to the data transfer node. It attempts to complete the data transfer at just the time of the pinned start time of jobs. The evaluation in this paper indicates that the proposed framework can keep data freshness high and minimize the job waiting time for data transfer.