Joint Memory Propagation and Rectification for Video Object Segmentation.

Spatio-temporal memory matching-based methods have achieved significant breakthroughs in the task of video object segmentation. Nevertheless, they are deficient in exploiting the long-term temporal correlation of frames by performing pixel-wise memory matching with limited reference frames, and tend to segment similar objects incorrectly. In this paper, we propose two novel modules, memory propagation and rectification modules, that are able to execute the memory reading in all proceeding frames while leveraging long-term temporal smoothness. Specifically, the memory propagation module employs the lightweight ConvGRU architecture to integrate abundant temporal cues from the reference frame sequence. Afterwards, the memory rectification module guided by the similarity between frames is proposed to increase the inter-frame consistency constraint to retrieve reliable memory and transfer them to the current frame. Additionally, we adopt a tiny-capacity spatio-temporal matching module to model the pixel relationship with only two reference frames. The experimental results on DAVIS and YouTube-VOS datasets demonstrate that combining memory propagation and rectification can achieve considerable performance gain and maintain constant inference time and memory usage in the sequence segmentation. To facilitate research in this direction, we release the project code at https://github.com/GDPUCJH/MPRNet .