X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
arxiv(2024)
摘要
Lately, there has been growing interest in adapting vision-language models
(VLMs) to image and third-person video classification due to their success in
zero-shot recognition. However, the adaptation of these models to egocentric
videos has been largely unexplored. To address this gap, we propose a simple
yet effective cross-modal adaptation framework, which we call X-MIC. Using a
video adapter, our pipeline learns to align frozen text embeddings to each
egocentric video directly in the shared embedding space. Our novel adapter
architecture retains and improves generalization of the pre-trained VLMs by
disentangling learnable temporal modeling and frozen visual encoder. This
results in an enhanced alignment of text embeddings to each egocentric video,
leading to a significant improvement in cross-dataset generalization. We
evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for
fine-grained cross-dataset action generalization, demonstrating the
effectiveness of our method. Code is available at
https://github.com/annusha/xmic
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要