EmoPoseFace: Head Pose Aware Speech-Driven 3D Emotional Facial Animation Using Latent Diffusion.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Generating realistic 3D facial animations for virtual reality (VR) applications like digital avatars faces significant challenges.
• Existing methods often fail to accurately synchronize lip movements, render natural expressions, and convey complex emotions simultaneously.
• The influence of head pose on enhancing facial emotional expressiveness remains underexplored.

Purpose of the Study:

• To develop a novel method for generating speech-driven 3D emotional facial animations that incorporate synchronized head poses.
• To address limitations in lip synchronization, natural expression, and emotional representation in current facial animation techniques.
• To investigate and leverage the impact of head pose on emotional expressiveness in 3D facial animation.

Main Methods:

• Proposed EmoPoseFace, a novel Diffusion-based network utilizing a dual-branch conditional generation architecture.
• Separately modeled facial expressions and head poses, integrating emotion and head-pose conditions for unified control.
• Introduced the Global-local Facial Fine-grained Editing Module (GL-FFE) for emotional enhancement and detailed facial modification.

Main Results:

• EmoPoseFace demonstrated superior performance over existing methods in lip-sync accuracy and emotional detail preservation.
• The integration of head pose control and the GL-FFE module significantly enhanced the expressiveness of virtual facial animations.
• User studies confirmed the effectiveness and naturalness of the fine-grained facial editing capabilities.

Conclusions:

• EmoPoseFace effectively generates speech-driven 3D emotional facial animations with synchronized head poses, overcoming limitations of previous methods.
• The proposed approach enhances emotional expressiveness and realism in virtual avatars through integrated head pose and fine-grained facial control.
• The method offers a significant advancement for creating more engaging and lifelike virtual characters in VR and related fields.