A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels
CoRR(2024)
摘要
Cross-modal retrieval (CMR) aims to establish interaction between different
modalities, among which supervised CMR is emerging due to its flexibility in
learning semantic category discrimination. Despite the remarkable performance
of previous supervised CMR methods, much of their success can be attributed to
the well-annotated data. However, even for unimodal data, precise annotation is
expensive and time-consuming, and it becomes more challenging with the
multimodal scenario. In practice, massive multimodal data are collected from
the Internet with coarse annotation, which inevitably introduces noisy labels.
Training with such misleading labels would bring two key challenges –
enforcing the multimodal samples to align incorrect semantics and
widen the heterogeneous gap, resulting in poor retrieval performance. To
tackle these challenges, this work proposes UOT-RCL, a Unified framework based
on Optimal Transport (OT) for Robust Cross-modal Retrieval. First, we propose a
semantic alignment based on partial OT to progressively correct the noisy
labels, where a novel cross-modal consistent cost function is designed to blend
different modalities and provide precise transport cost. Second, to narrow the
discrepancy in multi-modal data, an OT-based relation alignment is proposed to
infer the semantic-level cross-modal matching. Both of these two components
leverage the inherent correlation among multi-modal data to facilitate
effective cost function. The experiments on three widely-used cross-modal
retrieval datasets demonstrate that our UOT-RCL surpasses the state-of-the-art
approaches and significantly improves the robustness against noisy labels.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要