FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
CVPR 2024(2023)
Abstract
We introduce FaceTalk, a novel generative approach designed for synthesizing
high-fidelity 3D motion sequences of talking human heads from input audio
signal. To capture the expressive, detailed nature of human heads, including
hair, ears, and finer-scale eye movements, we propose to couple speech signal
with the latent space of neural parametric head models to create high-fidelity,
temporally coherent motion sequences. We propose a new latent diffusion model
for this task, operating in the expression space of neural parametric head
models, to synthesize audio-driven realistic head sequences. In the absence of
a dataset with corresponding NPHM expressions to audio, we optimize for these
correspondences to produce a dataset of temporally-optimized NPHM expressions
fit to audio-video recordings of people talking. To the best of our knowledge,
this is the first work to propose a generative approach for realistic and
high-quality motion synthesis of volumetric human heads, representing a
significant advancement in the field of audio-driven 3D animation. Notably, our
approach stands out in its ability to generate plausible motion sequences that
can produce high-fidelity head animation coupled with the NPHM shape space. Our
experimental results substantiate the effectiveness of FaceTalk, consistently
achieving superior and visually natural motion, encompassing diverse facial
expressions and styles, outperforming existing methods by 75
user study evaluation.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined