Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning
arxiv(2024)
摘要
Human-centered dynamic scene understanding plays a pivotal role in enhancing
the capability of robotic and autonomous systems, in which Video-based
Human-Object Interaction (V-HOI) detection is a crucial task in semantic scene
understanding, aimed at comprehensively understanding HOI relationships within
a video to benefit the behavioral decisions of mobile robots and autonomous
driving systems. Although previous V-HOI detection models have made significant
strides in accurate detection on specific datasets, they still lack the general
reasoning ability like human beings to effectively induce HOI relationships. In
this study, we propose V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), a
novel framework consisting of a series of plug-and-play modules that could
facilitate the performance of current V-HOI detection models by leveraging the
strong reasoning ability of different off-the-shelf pre-trained large language
models (LLMs). We design a two-stage collaboration system of different LLMs for
the V-HOI task. Specifically, in the first stage, we design a Cross-Agents
Reasoning scheme to leverage the LLM conduct reasoning from different aspects.
In the second stage, we perform Multi-LLMs Debate to get the final reasoning
answer based on the different knowledge in different LLMs. Additionally, we
devise an auxiliary training strategy that utilizes CLIP, a large
vision-language model to enhance the base V-HOI models' discriminative ability
to better cooperate with LLMs. We validate the superiority of our design by
demonstrating its effectiveness in improving the prediction accuracy of the
base V-HOI model via reasoning from multiple perspectives.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要