Emotionally expressive facial animation driven by EmotionBERT embeddings

Facial animation driven by speech signals plays a pivotal role in virtual reality, human-computer interaction, and entertainment. Traditional methods often struggle to capture the rich emotional and semantic context conveyed by the speaker's voice, resulting in less engaging animations. This paper introduces Speech2Face, a framework that synthesizes high-fidelity facial movements conditioned on expressive speech inputs. The proposed approach utilizes EmotionBERT to extract semantic and emotional embeddings from speech transcripts, enabling the generation of temporally coherent and contextually appropriate facial expressions. Experiments on VOCASET, IEMOCAP, MEAD, and BIWI datasets demonstrate that Speech2Face improves emotional accuracy by up to 80.8%, naturalness, and synchronization between speech and facial movements compared to existing models. This framework advances multi-modal emotion-aware synthesis, offering an effective solution for creating engaging and realistic digital humans. Our code is available at https://github.com/rizwanchouhan/speech2face.