High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering
CoRR(2024)
摘要
Readily editable mesh blendshapes have been widely used in animation
pipelines, while recent advancements in neural geometry and appearance
representations have enabled high-quality inverse rendering. Building upon
these observations, we introduce a novel technique that reconstructs mesh-based
blendshape rigs from single or sparse multi-view videos, leveraging
state-of-the-art neural inverse rendering. We begin by constructing a
deformation representation that parameterizes vertex displacements into
differential coordinates with tetrahedral connections, allowing for
high-quality vertex deformation on high-resolution meshes. By constructing a
set of semantic regulations in this representation, we achieve joint
optimization of blendshapes and expression coefficients. Furthermore, to enable
a user-friendly multi-view setup with unsynchronized cameras, we propose a
neural regressor to model time-varying motion parameters. This approach
implicitly considers the time difference across multiple cameras, enhancing the
accuracy of motion modeling. Experiments demonstrate that, with the flexible
input of single or sparse multi-view videos, we reconstruct personalized
high-fidelity blendshapes. These blendshapes are both geometrically and
semantically accurate, and they are compatible with industrial animation
pipelines. Code and data will be released.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要