Towards Lightweight Deep Reference Frame for Versatile Video Coding.

Deep neural network (DNN)-based methods have demonstrated enormous potential for Versatile Video Coding (VVC) inter prediction enhancement. However, due to their typically high computational complexity, implementing them in practical applications can be challenging. In this paper, we propose a lightweight deep reference frame interpolation network to enhance bi-prediction with low complexity. Specifically, given a pair of bi-directional reconstructed frames, first, we down-sample input frames to reduce the complexity before feeding them into the optical flow estimation network. Then the optical flows are utilized to warp extracted features at three different levels. The warped features are fused to generate the output intermediate frame. The additional reference frame is inserted into the reference picture lists to provide an additional reliable reference candidate. In contrast to previous efforts, the proposed method aims at achieving the trade-off between performance and complexity while maintaining a complexity of about 64 kMACs/pix. Experimental results demonstrate that our method achieves 1.82%/2.43%/2.02% coding efficiency improvements for Y/U/V components under random access (RA) configuration compared to the latest NNVC standard software VTM-11.0_NNVC-5.0.