AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
arxiv(2023)
摘要
Audio-visual representation learning aims to develop systems with human-like
perception by utilizing correlation between auditory and visual information.
However, current models often focus on a limited set of tasks, and
generalization abilities of learned representations are unclear. To this end,
we propose the AV-SUPERB benchmark that enables general-purpose evaluation of
unimodal audio/visual and bimodal fusion representations on 7 datasets covering
5 audio-visual tasks in speech and audio processing. We evaluate 5 recent
self-supervised models and show that none of these models generalize to all
tasks, emphasizing the need for future study on improving universal model
performance. In addition, we show that representations may be improved with
intermediate-task fine-tuning and audio event classification with AudioSet
serves as a strong intermediate task. We release our benchmark with evaluation
code and a model submission platform to encourage further research in
audio-visual learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要