Perceptual rate-distortion optimization for H.264/AVC video coding from both signal and vision perspectives

Recent researches have shown that the Structural SIMilariy (SSIM)-based Rate-Distortion Optimization (RDO) can obtain more structural information than the traditional SSE-based RDO for video coding. Correspondingly, the perceptual video quality is improved at certain degree for the video stream encoded by the SSIM-based RDO. However, for the MB with significant luminance change (due to the flashlight or sunshine change, etc.) but little structure change compared to the co-located MB in the referred frame, the SSIM-based RDO may select improper encoding mode (SKIP) which leads to uncomfortable visual experience. In this paper, involving the surrounding pixels of the current coding MB into the measurement of spacial texture quality degradation, the RDO which jointly considers the SSIM-based distortion and SSE-based distortion is proposed to improve H.264/AVC perceptual coding performance. The Lagrange multiplier in the proposed RDO is firstly derived at the frame level. Then, to make the Lagrange multiplier more adaptive to the specific video content, the Lagrange multiplier for each individual MB is refined based on the visual information theory. Experimental results show that the proposed perceptual RDO can select the optimal encoding mode which preserves as much structural information as possible with as little SSE-based distortion as possible, and that the perceptual quality of the encoded video is improved.