Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
CoRR(2024)
摘要
One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from
an unseen image, and then animate it with a reference video or audio to
generate a talking portrait video. The existing methods fail to simultaneously
achieve the goals of accurate 3D avatar reconstruction and stable talking face
animation. Besides, while the existing works mainly focus on synthesizing the
head part, it is also vital to generate natural torso and background segments
to obtain a realistic talking portrait video. To address these limitations, we
present Real3D-Potrait, a framework that (1) improves the one-shot 3D
reconstruction power with a large image-to-plane model that distills 3D prior
knowledge from a 3D face generative model; (2) facilitates accurate
motion-conditioned animation with an efficient motion adapter; (3) synthesizes
realistic video with natural torso movement and switchable background using a
head-torso-background super-resolution model; and (4) supports one-shot
audio-driven talking face generation with a generalizable audio-to-motion
model. Extensive experiments show that Real3D-Portrait generalizes well to
unseen identities and generates more realistic talking portrait videos compared
to previous methods.
更多查看译文
关键词
Neural Radiance Field,One-shot Talking Face Generation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要