Eye vs. AI: Human Gaze and Model Attention in Video Memorability
CoRR(2023)
摘要
Understanding the factors that determine video memorability has important
applications in areas such as educational technology and advertising. Towards
this goal, we investigate the semantic and temporal attention mechanisms
underlying video memorability. We propose a Transformer-based model with
spatio-temporal attention that matches SoTA performance on video memorability
prediction on a large naturalistic video dataset. More importantly, the
self-attention patterns show us where the model looks to predict memorability.
We compare model attention against human gaze fixation density maps collected
through a small-scale eye-tracking experiment where humans perform a video
memory task. Quantitative saliency metrics show that the model attention and
human gaze follow similar patterns. Furthermore, while panoptic segmentation
confirms that the model and humans attend more to thing classes, stuff classes
that receive increased/decreased attention tend to have higher memorability
scores. We also observe that the model assigns greater importance to the
initial frames, mimicking temporal attention patterns found in humans.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要