Reinforcement learning for video encoder control in HEVC

In todays video compression systems, the encoder typically follows an optimization procedure to find a compressed representation of the video signal. While primary optimization criteria are bit rate and image distortion, low complexity of this procedure may also be of importance in some applications, making complexity a third objective. We approach this problem by treating the encoding procedure as a decision process in time and make it amenable to reinforcement learning. Our learning algorithm computes a strategy in a compact functional representation, which is then employed in the video encoder to control its search. By including measured execution time into the reinforcement signal with a lagrangian weight, we realize a trade-off between RD-performance and computational complexity controlled by a single parameter. Using the reference software test model (HM) of the HEVC video coding standard, we show that over half the encoding time can be saved at the same RD-performance.