Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
arxiv(2023)
摘要
We investigate compositional structures in data embeddings from pre-trained
vision-language models (VLMs). Traditionally, compositionality has been
associated with algebraic operations on embeddings of words from a pre-existing
vocabulary. In contrast, we seek to approximate representations from an encoder
as combinations of a smaller set of vectors in the embedding space. These
vectors can be seen as "ideal words" for generating concepts directly within
the embedding space of the model. We first present a framework for
understanding compositional structures from a geometric perspective. We then
explain what these compositional structures entail probabilistically in the
case of VLM embeddings, providing intuitions for why they arise in practice.
Finally, we empirically explore these structures in CLIP's embeddings and we
evaluate their usefulness for solving different vision-language tasks such as
classification, debiasing, and retrieval. Our results show that simple linear
algebraic operations on embedding vectors can be used as compositional and
interpretable methods for regulating the behavior of VLMs.
更多查看译文
关键词
compositional structures,meanings,models,vision-language
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要