Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion.

Area of Science:

• Computer Graphics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• 3D facial animations are vital for augmented and mixed reality (AR/MR) but struggle with authentic emotional expression.
• Existing methods for facial animation lack the nuance required for genuine emotional representation.

Purpose of the Study:

• To develop a novel approach for capturing fine facial expressions and generating authentic, audio-synchronized 3D facial animations.
• To enhance the emotional depth and immersive quality of AR/MR applications through realistic facial animation.

Main Methods:

• Introduced a Local-to-global Latent Diffusion Model (LG-LDM) integrating audio, time, and expression conditions for encoding emotionally rich features.
• Developed a Facial Denoiser Model (FDM) within LG-LDM to align local-to-global animation features with audio signals.
• Redesigned an Emotion-centric Vector Quantized-Variational AutoEncoder (EVQ-VAE) for precise decoding of subtle emotional differences and 3D facial geometry reconstruction.

Main Results:

• The proposed method effectively generates emotionally realistic 3D facial animations synchronized with audio inputs.
• LG-LDM and EVQ-VAE successfully capture and reconstruct subtle facial expressions, improving animation authenticity.
• The approach addresses key challenges in audio-driven, emotionally realistic facial animation.

Conclusions:

• This work significantly advances the state-of-the-art in generating emotionally realistic 3D facial animations for AR/MR.
• The developed models enhance the immersive experience and emotional depth in digital media applications.
• A reproducibility kit with code and data is provided to facilitate further research.