A Modular Approach for Multimodal Summarization of TV Shows
arxiv(2024)
摘要
In this paper we address the task of summarizing television shows, which
touches key areas in AI research: complex reasoning, multiple modalities, and
long narratives. We present a modular approach where separate components
perform specialized sub-tasks which we argue affords greater flexibility
compared to end-to-end methods. Our modules involve detecting scene boundaries,
reordering scenes so as to minimize the number of cuts between different
events, converting visual information to text, summarizing the dialogue in each
scene, and fusing the scene summaries into a final summary for the entire
episode. We also present a new metric, PREFS (Precision and
Recall Evaluation of Summary Facts), to
measure both precision and recall of generated summaries, which we decompose
into atomic facts. Tested on the recently released SummScreen3D dataset
Papalampidi and Lapata (2023), our method produces higher quality summaries
than comparison models, as measured with ROUGE and our new fact-based metric.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要