Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models
CoRR(2024)
摘要
Recent advancements in Multimodal Large Language Models (MLLMs) have
significantly enhanced the comprehension of multimedia content, bringing
together diverse modalities such as text, images, and videos. However, a
critical challenge faced by these models, especially when processing video
inputs, is the occurrence of hallucinations - erroneous perceptions or
interpretations, particularly at the event level. This study introduces an
innovative method to address event-level hallucinations in MLLMs, focusing on
specific temporal understanding in video content. Our approach leverages a
novel framework that extracts and utilizes event-specific information from both
the event query and the provided video to refine MLLMs' response. We propose a
unique mechanism that decomposes on-demand event queries into iconic actions.
Subsequently, we employ models like CLIP and BLIP2 to predict specific
timestamps for event occurrences. Our evaluation, conducted using the
Charades-STA dataset, demonstrates a significant reduction in temporal
hallucinations and an improvement in the quality of event-related responses.
This research not only provides a new perspective in addressing a critical
limitation of MLLMs but also contributes a quantitatively measurable method for
evaluating MLLMs in the context of temporal-related questions.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要