What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
arxiv(2023)
摘要
Counterfactual reasoning, a fundamental aspect of human cognition, involves
contemplating alternatives to established facts or past events, significantly
enhancing our abilities in planning and decision-making. In light of the
advancements in current multi-modal large language models, we explore their
effectiveness in counterfactual reasoning. To facilitate this investigation, we
introduce a novel dataset, C-VQA, specifically designed to test the
counterfactual reasoning capabilities of modern multi-modal large language
models. This dataset is constructed by infusing original questions with
counterfactual presuppositions, spanning various types such as numerical and
boolean queries. It encompasses a mix of real and synthetic data, representing
a wide range of difficulty levels. Our thorough evaluations of contemporary
vision-language models using this dataset have revealed substantial performance
drops, with some models showing up to a 40
significant gap between current models and human-like vision reasoning
capabilities. We hope our dataset will serve as a vital benchmark for
evaluating the counterfactual reasoning capabilities of models. Code and
dataset are publicly available at https://bzhao.me/C-VQA/.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要