Emotionally expressive facial animation driven by EmotionBERT embeddings.

Area of Science:

• Computer Vision
• Natural Language Processing
• Affective Computing

Background:

• Facial animation driven by speech is crucial for virtual reality (VR), human-computer interaction (HCI), and entertainment.
• Existing methods often fail to capture the full emotional and semantic nuances of speech, leading to less immersive experiences.

Purpose of the Study:

• To introduce Speech2Face, a framework for synthesizing high-fidelity facial movements directly from expressive speech.
• To improve the emotional accuracy, naturalness, and synchronization of speech-driven facial animation.

Main Methods:

• Utilized EmotionBERT for extracting semantic and emotional embeddings from speech transcripts.
• Developed a framework for generating temporally coherent and contextually appropriate facial expressions conditioned on speech inputs.
• Evaluated performance on VOCASET, IEMOCAP, MEAD, and BIWI datasets.

Main Results:

• Achieved significant improvements in emotional accuracy, with gains up to 80.8% compared to existing models.
• Demonstrated enhanced naturalness and synchronization between synthesized facial movements and speech.
• Validated the framework's effectiveness across multiple benchmark datasets.

Conclusions:

• Speech2Face offers an effective solution for multi-modal emotion-aware synthesis.
• The framework advances the creation of engaging and realistic digital humans by better capturing vocal expressivity.