Teaching Text-to-Image Models to Communicate in Dialog
CoRR(2023)
摘要
A picture is worth a thousand words, thus, it is crucial for conversational
agents to understand, perceive, and effectively respond with pictures. However,
we find that directly employing conventional image generation techniques is
inadequate for conversational agents to produce image responses effectively. In
this paper, we focus on the innovative dialog-to-image generation task, where
the model synthesizes a high-resolution image aligned with the given dialog
context as a response. To tackle this problem, we design a tailored fine-tuning
approach on the top of state-of-the-art text-to-image generation models to
fully exploit the structural and semantic features in dialog context during
image generation. Concretely, we linearize the dialog context with specific
indicators to maintain the dialog structure, and employ in-domain data to
alleviate the style mismatch between dialog-to-image and conventional image
generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that
our approach brings consistent and remarkable improvement with 3
state-of-the-art pre-trained text-to-image generation backbones.
更多查看译文
关键词
models,communicate,text-to-image
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要