Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
CoRR(2024)
摘要
The evaluation of text-generative vision-language models is a challenging yet
crucial endeavor. By addressing the limitations of existing Visual Question
Answering (VQA) benchmarks and proposing innovative evaluation methodologies,
our research seeks to advance our understanding of these models' capabilities.
We propose a novel VQA benchmark based on well-known visual classification
datasets which allows a granular evaluation of text-generative vision-language
models and their comparison with discriminative vision-language models. To
improve the assessment of coarse answers on fine-grained classification tasks,
we suggest using the semantic hierarchy of the label space to ask automatically
generated follow-up questions about the ground-truth category. Finally, we
compare traditional NLP and LLM-based metrics for the problem of evaluating
model predictions given ground-truth answers. We perform a human evaluation
study upon which we base our decision on the final metric. We apply our
benchmark to a suite of vision-language models and show a detailed comparison
of their abilities on object, action, and attribute classification. Our
contributions aim to lay the foundation for more precise and meaningful
assessments, facilitating targeted progress in the exciting field of
vision-language modeling.
更多查看译文
关键词
Open-ended VQA,benchmark,Vision-Language,VL,Vision-Text,VLM,Vision-Language models,Image classification,Visual question answering,Text-generating VLM
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要