Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
ICLR 2024(2024)
摘要
Video diffusion models have recently made great progress in generation
quality, but are still limited by the high memory and computational
requirements. This is because current video diffusion models often attempt to
process high-dimensional videos directly. To tackle this issue, we propose
content-motion latent diffusion model (CMD), a novel efficient extension of
pretrained image diffusion models for video generation. Specifically, we
propose an autoencoder that succinctly encodes a video as a combination of a
content frame (like an image) and a low-dimensional motion latent
representation. The former represents the common content, and the latter
represents the underlying motion in the video, respectively. We generate the
content frame by fine-tuning a pretrained image diffusion model, and we
generate the motion latent representation by training a new lightweight
diffusion model. A key innovation here is the design of a compact latent space
that can directly utilizes a pretrained image diffusion model, which has not
been done in previous latent video diffusion models. This leads to considerably
better quality generation and reduced computational costs. For instance, CMD
can sample a video 7.7× faster than prior approaches by generating a
video of 512×1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
achieves an FVD score of 212.7 on WebVid-10M, 27.3
state-of-the-art of 292.4.
更多查看译文
关键词
video generation,diffusion models
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要