Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination
CoRR(2024)
摘要
The potential of Vision-Language Models (vlms) often remains
underutilized in handling complex text-based problems, particularly when these
problems could benefit from visual representation. Resonating with humans'
ability to solve complex text-based problems by (1) creating a visual diagram
from the problem and (2) deducing what steps they need to take to solve it, we
propose Self-Imagine. We leverage a single Vision-Language Model
(vlm) to generate a structured representation of the question using
HTML, then render the HTML as an image, and finally use the same to answer
the question using both the question and the image. Our approach does not
require any additional training data or training. We evaluate our approach in
three mathematics tasks and nine general-purpose reasoning tasks using
state-of-the-art vlm. Our approach boosts the performance of
vlm on all math tasks (: +4.62%; : +4.49%; :
+9.30%) and the majority of the general-purpose reasoning tasks by 0.4% to
13.20% while achieving comparable performance in other tasks.
Code and data at https://github.com/snat1505027/self-imagine .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要