AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
ICLR 2024(2023)
摘要
With the advance of text-to-image (T2I) diffusion models (e.g., Stable
Diffusion) and corresponding personalization techniques such as DreamBooth and
LoRA, everyone can manifest their imagination into high-quality images at an
affordable cost. However, adding motion dynamics to existing high-quality
personalized T2Is and enabling them to generate animations remains an open
challenge. In this paper, we present AnimateDiff, a practical framework for
animating personalized T2I models without requiring model-specific tuning. At
the core of our framework is a plug-and-play motion module that can be trained
once and seamlessly integrated into any personalized T2Is originating from the
same base T2I. Through our proposed training strategy, the motion module
effectively learns transferable motion priors from real-world videos. Once
trained, the motion module can be inserted into a personalized T2I model to
form a personalized animation generator. We further propose MotionLoRA, a
lightweight fine-tuning technique for AnimateDiff that enables a pre-trained
motion module to adapt to new motion patterns, such as different shot types, at
a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA
on several public representative personalized T2I models collected from the
community. The results demonstrate that our approaches help these models
generate temporally smooth animation clips while preserving the visual quality
and motion diversity. Codes and pre-trained weights are available at
https://github.com/guoyww/AnimateDiff.
更多查看译文
关键词
Deep Learning,Diffusion Model,Video Generation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要