Joint Asymptotic Closed-Loop Design of Secondary Transform and Scan Order for Inter Coding in AV1

Most video coding systems employ separable transforms due to their low computational complexity and storage requirements, and despite their sub-optimal decorrelating capabilities. To achieve better decorrelation, recent codecs further apply a non-separable secondary transform to low-frequency primary transform coefficients of the intra-prediction residual. This paper focuses on effective design of non-separable secondary transforms for the inter-prediction residual. As the combination of primary and secondary transforms yields a set of ultimate transform coefficients for which the default zig-zag scan order is sub-optimal, we complement the secondary transform design with the design of corresponding coefficient scanning order modes that facilitate effective entropy coding. A critical stability challenge in this joint design, due to error propagation through the codec's prediction loop, is circumvented by leveraging the asymptotic closed loop (ACL) design paradigm. ACL operates in open-loop in each iteration to ensure stability, but in a manner that ultimately converges to parameter optimization for closed-loop operation. Experimental results show an average gains in BD rate of 0.76% for CIF and 0.41% for HD sequences, over the AV1 codec (libaom).