Semantically Disentangled Variational Autoencoder for Modeling 3D Facial Details

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Parametric face models (morphable, blendshape) excel at large-scale facial geometry but lack detail parameterization (e.g., wrinkles).
• This limitation hinders accuracy and realism in face representation, reconstruction, and animation.

Purpose of the Study:

• To develop a method for parameterizing and independently manipulating facial details, extending existing large-scale face models.
• To improve the accuracy and realism of 3D face reconstruction and animation by modeling fine-grained facial features.

Main Methods:

• Proposed a Semantically Disentangled Variational Autoencoder (SDVAE) using Deep Neural Networks for detail modeling.
• Employed adversarial training to disentangle semantic factors (identity, expression, age), eliminating correlations.
• Integrated the SDVAE as an extension to off-the-shelf large-scale face models.

Main Results:

• Achieved accurate and robust 3D face reconstruction from scans and images.
• Demonstrated independent control and animation of wrinkle-level details across various identities, expressions, and ages.
• Showcased superior performance in practical applications like scan/image fitting, video tracking, and model manipulation.

Conclusions:

• The proposed SDVAE effectively models facial details, offering greater accuracy and representation power than linear models.
• Enables independent manipulation and animation of semantic facial details, significantly enhancing realism.
• Provides a robust and versatile solution for 3D face reconstruction and detail control.