Efficient Data Loading for Deep Neural Network Training.

Abstract Deep neural networks (DNN) have achieved outstanding results in a wide range of applications. However, deep neural network training is a data-intensive and compute-intensive task to obtain high accuracy. While multiple state-of-the-art AI accelerators such as GPUs can be deployed in a single machine for training, data loading for deep neural network training dominates a significant amount of time in the whole training process. As a result, data loading becomes the key bottleneck to limit the whole training performance. This issue is mainly caused by the factor that a large number of small-size files are loaded during the training process. In this paper, we present efficient data loading for DNN training by using two methods: Based on data access pattern and efficient utilization of disk I/O, we use a block as the unit for data reading. As data are repeated used in training an accurate model, we cache partially loaded data for faster access. We use image-net as the dataset and present comprehensive evaluations in terms of running time and accuracy for stateof-the-art deep neural networks on a machine with four V100 GPU cards. The evaluation results demonstrated while that the accuracy is maintained for commonly used DNNs, our optimizations can achieve up to 10 fold performance growth in terms of the whole running time for DNN training.