Image-Caption Encoding for Improving Zero-Shot Generalization
CoRR(2024)
摘要
Recent advances in vision-language models have combined contrastive
approaches with generative methods to achieve state-of-the-art (SOTA) on
downstream inference tasks like zero-shot image classification. However, a
persistent issue of these models for image classification is their
out-of-distribution (OOD) generalization capabilities. We first show that when
an OOD data point is misclassified, the correct class can be typically found in
the Top-K predicted classes. In order to steer the model prediction toward the
correct class within the top predicted classes, we propose the Image-Caption
Encoding (ICE) method, a straightforward approach that directly enforces
consistency between the image-conditioned and caption-conditioned predictions
at evaluation time only. Intuitively, we take advantage of unique properties of
the generated captions to guide our local search for the correct class label
within the Top-K predicted classes. We show that our method can be easily
combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5
average and up to 3
https://github.com/Chris210634/ice
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要