Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
NeurIPS(2023)
摘要
Text-conditioned image generation models often generate incorrect
associations between entities and their visual attributes. This reflects an
impaired mapping between linguistic binding of entities and modifiers in the
prompt and visual binding of the corresponding elements in the generated image.
As one notable example, a query like "a pink sunflower and a yellow flamingo"
may incorrectly produce an image of a yellow sunflower and a pink flamingo. To
remedy this issue, we propose SynGen, an approach which first syntactically
analyses the prompt to identify entities and their modifiers, and then uses a
novel loss function that encourages the cross-attention maps to agree with the
linguistic binding reflected by the syntax. Specifically, we encourage large
overlap between attention maps of entities and their modifiers, and small
overlap with other entities and modifier words. The loss is optimized during
inference, without retraining or fine-tuning the model. Human evaluation on
three datasets, including one new and challenging set, demonstrate significant
improvements of SynGen compared with current state of the art methods. This
work highlights how making use of sentence structure during inference can
efficiently and substantially improve the faithfulness of text-to-image
generation.
更多查看译文
关键词
attention map alignment,diffusion models,attribute
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要