Detours for Navigating Instructional Videos
CoRR(2024)
摘要
We introduce the video detours problem for navigating instructional videos.
Given a source video and a natural language query asking to alter the how-to
video's current path of execution in a certain way, the goal is to find a
related ”detour video” that satisfies the requested alteration. To address
this challenge, we propose VidDetours, a novel video-language approach that
learns to retrieve the targeted temporal segments from a large repository of
how-to's using video-and-text conditioned queries. Furthermore, we devise a
language-based pipeline that exploits how-to video narration text to create
weakly supervised training data. We demonstrate our idea applied to the domain
of how-to cooking videos, where a user can detour from their current recipe to
find steps with alternate ingredients, tools, and techniques. Validating on a
ground truth annotated dataset of 16K samples, we show our model's significant
improvements over best available methods for video retrieval and question
answering, with recall rates exceeding the state of the art by 35
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要