Video Super-Resolution With Phase-Aided Deformable Alignment Network

Video super-resolution (VSR) aims to restore a high-resolution frame from its corresponding reference frame and a series of neighboring frames in low resolution. Because of the displacement between the reference frame and each neighboring frame, which is caused by the motion of the camera or observed objects, VSR methods usually initially align the neighboring frames with the reference frame. However, the commonly used motion estimation and compensation methods highly depend on the predicted optical flow and they are affected by the lighting change. We propose a phase-aided deformable alignment network (PDAN) to alleviate the above problems. In PDAN, a phase-based method is introduced in the motion estimation subnetwork along with the traditional U-Net structure, which improves the restoration robustness in scenarios of lighting change. A deformable convolutional network is further adopted in the alignment subnetwork to enhance the alignment for objects with an irregular shape and motion distortion, which also avoids the dependence on explicit motion representation accuracy. Moreover, the reconstruction module is optimized for improved restoration. Extensive experiments conducted demonstrate that PDAN achieves state-of-the-art quantitative and qualitative performances. (C) 2020 SPIE and IS&T