Multi-Modal Mutual Topic Reinforce Modeling For Cross-Media Retrieval
MM(2014)
摘要
As an important and challenging problem in the multimedia area, multi-modal data understanding aims to explore the intrinsic semantic information across different modalities in a collaborative manner. To address this problem, a possible solution is to effectively and adaptively capture the common cross-modal semantic information by modeling the inherent correlations between the latent topics from different modalities. Motivated by this task, we propose a supervised multi-modal mutual topic reinforce modeling ((MR)-R-3) approach, which seeks to build a joint cross-modal probabilistic graphical model for discovering the mutually consistent semantic topics via appropriate interactions between model factors (e.g., categories, latent topics and observed multi-modal data). In principle, (MR)-R-3 is capable of simultaneously accomplishing the following two learning tasks: 1) modality-specific (e.g., image-specific or text-specific) latent topic learning; and 2) cross-modal mutual topic consistency learning. By investigating the cross-modal topic-related distribution information, (MR)-R-3 encourages to disentangle the semantically consistent cross-modal topics (containing some common semantic information across different mod all ties). In other words, the semantically co-occurring cross-modal topics are reinforced by (MR)-R-3 through adaptively passing the mutually reinforced messages to each other in the model-learning process. To further enhance the discriminative power of the learned latent topic representations, (MR)-R-3 incorporates the auxiliary information (i.e., categories or labels) into the process of Bayesian modeling, which boosts the modeling capability of capturing the inter-class discriminative information. Experimental results over two benchmark datasets demonstrate the effectiveness of the proposed (MR)-R-3 in cross-modal retrieval.
更多查看译文
关键词
Multi-modal Analysis,Mutual Topic,Topic Reinforcement,Cross-media Retrieval
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络