Training-Free Semantic Video Composition via Pre-trained Diffusion Model
CoRR(2024)
摘要
The video composition task aims to integrate specified foregrounds and
backgrounds from different videos into a harmonious composite. Current
approaches, predominantly trained on videos with adjusted foreground color and
lighting, struggle to address deep semantic disparities beyond superficial
adjustments, such as domain gaps. Therefore, we propose a training-free
pipeline employing a pre-trained diffusion model imbued with semantic prior
knowledge, which can process composite videos with broader semantic
disparities. Specifically, we process the video frames in a cascading manner
and handle each frame in two processes with the diffusion model. In the
inversion process, we propose Balanced Partial Inversion to obtain generation
initial points that balance reversibility and modifiability. Then, in the
generation process, we further propose Inter-Frame Augmented attention to
augment foreground continuity across frames. Experimental results reveal that
our pipeline successfully ensures the visual harmony and inter-frame coherence
of the outputs, demonstrating efficacy in managing broader semantic
disparities.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要