ooc_cuDNN: Accommodating convolutional neural networks over GPU memory capacity

GPUs are widely used to accelerate deep learning with convolutional neural networks (CNNs). However, since GPU memory capacity is limited, it is difficult to implement efficient programs that compute large CNNs on GPU. This paper describes the design and implementation of out-of-core cuDNN (ooc cuDNN) library. It supports to compute CNNs exceeding GPU memory capacity using capacity of CPU memory. ooc cuDNN is an extension of cuDNN, which is high performance and popular deep learning library for GPUs. ooc cuDNN divides CNN computation based on its performance model for performance improvement. In addition, ooc cuDNN provides fused functions, which are combinations of several kernel functions to reduce extra communication costs. With ooc cuDNN, we successfully computed CNN requiring more than 60 GB memory on a single GPU with 16 GB memory. Compared with an in-core case using cuDNN, performance degradation was 13%.