Audio-driven single image talking face animation with transformers.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Audio-driven talking-head video generation is vital for virtual humans and digital content.
• Current methods struggle with unnatural lip movements and facial distortions, especially during emotional expressions.
• These issues stem from audio signals entangling linguistic content, emotion, and speaker attributes.

Purpose of the Study:

• To propose ExpNet, a novel framework for realistic audio-driven talking-head video generation.
• To address limitations in existing methods regarding expression realism and lip synchronization.
• To enhance the quality of generated videos, particularly under exaggerated expressions and emotional variations.

Main Methods:

• Utilized a Transformer-based expression regression framework (ExpNet) employing 3DMM coefficients.
• Implemented a conditional Variational Autoencoder (VAE) for head pose coefficient generation.
• Employed a CNN-Transformer architecture with ALiBi-based relative positional bias for expression coefficient regression, conditioned on the first frame.

Main Results:

• ExpNet demonstrated superior performance over existing methods in expression realism, lip synchronization, and video quality across multiple datasets (HDTF, MEAD, LRS3).
• Ablation studies confirmed the importance of ALiBi, landmark supervision, and the Transformer module for temporal stability and reduced lip jitter.
• The framework successfully preserved identity and emotion consistency throughout the generated videos.

Conclusions:

• ExpNet effectively generates high-quality talking-head videos by decoupling head motion and facial expressions.
• The proposed method offers significant improvements in realism and synchronization compared to prior approaches.
• Key components like ALiBi and Transformer modules are critical for robust and consistent facial animation.