Cross-modal fusion encoder via graph neural network for referring image segmentation
IET IMAGE PROCESSING(2024)
摘要
Referring image segmentation identifies the object masks from images with the guidance of input natural language expressions. Nowadays, many remarkable cross-modal decoder are devoted to this task. But there are mainly two key challenges in these models. One is that these models usually lack to extract fine-grained boundary information and gradient information of images. The other is that these models usually lack to explore language associations among image pixels. In this work, a Multi-scale Gradient balanced Central Difference Convolution (MG-CDC) and a Graph convolutional network-based Language and Image Fusion (GLIF) for cross-modal encoder, called Graph-RefSeg, are designed. Specifically, in the shallow layer of the encoder, the MG-CDC captures comprehensive fine-grained image features. It could enhance the perception of target boundaries and provide effective guidance for deeper encoding layers. In each encoder layer, the GLIF is used for cross-modal fusion. It could explore the correlation of every pixel and its corresponding language vectors by a graph neural network. Since the encoder achieves robust cross-modal alignment and context mining, a light-weight decoder could be used for segmentation prediction. Extensive experiments show that the proposed Graph-RefSeg outperforms the state-of-the-art methods on three public datasets. Code and models will be made publicly available at . In this work, we design a Multi-scale Gradient balanced Central Difference Convolution (MG-CDC) and a Graph convolutional network-based Language and Image Fusion (GLIF) for cross-modal encoder. Since our encoder achieves robust cross-modal alignment and context mining, we could use a light-weight decoder for segmentation prediction. Extensive experiments show that our method outperforms the state-of-the-art methods on three public datasets.image
更多查看译文
关键词
image fusion,image segmentation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要