Bagged Tree And ResNet Based Joint End-to-End Fast CTU Partition Decision Algorithm For Video Intra Coding

In the development of video coding standards, advanced ones have greatly improved the bit rate compared with those of previous generation, but also brought a huge increase in coding complexity. Coding standards, such as high efficiency video coding (HEVC), versatile video coding (VVC) and AOMedia video 2 (AV2), get the optimal encoding performance by traversing all possible combinations of coding unit (CU) partition and selecting the combination with minimum coding cost. This process of searching for the best makes up a large part of encoding complexity. To reduce the complexity of coding block partition for many video coding standards, this paper proposes an end-to-end fast algorithm for partition structure decision of coding tree unit (CTU) in intra coding. It can be extended to various coding standards with fine tuning, and is applied to the intra coding of HEVC reference software HM16.7 as an example. In the proposed method, the splitting decision of a CTU is made by a well designed bagged tree model firstly. Then, the partition problem of a 32×32 sized CU is modeled as a 17-output classification task and solved by a well trained residual network (ResNet). Jointly using bagged tree and ResNet, the proposed fast CTU partition algorithm is able to generate the partition quad-tree structure of a CTU through an end-to-end prediction process, instead of multiple decision making procedures at depth level. Besides, several effective and representative datasets are also conducted in this paper to lay the foundation of high prediction accuracy. Compared with the original HM16.7 encoder, experimental results show that the proposed algorithm can reduce the encoding time by 59.79% on average, while the BD-rate loss is as less as 2.02%, which outperforms the results of most of state-of-the-art approaches in the fast intra CU partition area.