SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
CoRR(2023)
摘要
The development of text-to-video (T2V), i.e., generating videos with a given
text prompt, has been significantly advanced in recent years. However, relying
solely on text prompts often results in ambiguous frame composition due to
spatial uncertainty. The research community thus leverages the dense structure
signals, e.g., per-frame depth/edge sequences, to enhance controllability,
whose collection accordingly increases the burden of inference. In this work,
we present SparseCtrl to enable flexible structure control with temporally
sparse signals, requiring only one or a few inputs, as shown in Figure 1. It
incorporates an additional condition encoder to process these sparse signals
while leaving the pre-trained T2V model untouched. The proposed approach is
compatible with various modalities, including sketches, depth maps, and RGB
images, providing more practical control for video generation and promoting
applications such as storyboarding, depth rendering, keyframe animation, and
interpolation. Extensive experiments demonstrate the generalization of
SparseCtrl on both original and personalized T2V generators. Codes and models
will be publicly available at https://guoyww.github.io/projects/SparseCtrl .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要