Towards Next Generation Video Coding: from Neural Network Based Predictive Coding to In-Loop Filtering

Audio Video Coding Standard (AVS) Intelligent Coding Group mainly studies video coding tools based on neural network technology and its potential benefit for next generation video coding. Extensive efforts have been dedicated to the research on neural network (NN) based coding tools. In this paper, we present a novel NN based video coding framework by leveraging the supervised trained NN models for multiple modules in the hybrid coding framework, from the predictive coding to the in-loop filtering. Specifically, NN based intra prediction models the non-linear mapping from contextual pixels to the predictions. The inter prediction efficiency is enhanced by introducing a virtual reference frame (VRF) network. The convolutional neural network based loop filtering (CNNLF) with discriminative model selection exploits the texture adaptivity. The experimental results show that the CNNLF, NN Intra, and VRF models can bring 8.60%, 1.02%, and 2.26% luma BD-rate reduction under random access (RA) configuration compared with AVS reference software HPM13.0. Additional experiments with the combined three NN coding tools reveal that around 13% YUV BD-rate reduction could be obtained. The proposed framework opens novel sights for next generation video coding from the intelligent coding perspective.