StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
arxiv(2024)
摘要
Story visualization aims to generate a series of realistic and coherent
images based on a storyline. Current models adopt a frame-by-frame architecture
by transforming the pre-trained text-to-image model into an auto-regressive
manner. Although these models have shown notable progress, there are still
three flaws. 1) The unidirectional generation of auto-regressive manner
restricts the usability in many scenarios. 2) The additional introduced story
history encoders bring an extremely high computational cost. 3) The story
visualization and continuation models are trained and inferred independently,
which is not user-friendly. To these ends, we propose a bidirectional, unified,
and efficient framework, namely StoryImager. The StoryImager enhances the
storyboard generative ability inherited from the pre-trained text-to-image
model for a bidirectional generation. Specifically, we introduce a Target Frame
Masking Strategy to extend and unify different story image generation tasks.
Furthermore, we propose a Frame-Story Cross Attention Module that decomposes
the cross attention for local fidelity and global coherence. Moreover, we
design a Contextual Feature Extractor to extract contextual information from
the whole storyline. The extensive experimental results demonstrate the
excellent performance of our StoryImager. The code is available at
https://github.com/tobran/StoryImager.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要