Decoupling upper and lower face transformers for binary interactive video generation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Processing

Background:

• Existing audio-driven methods struggle with the complex relationship between speech and facial dynamics.
• Accurately synchronizing facial expressions with audio input remains a significant challenge in video generation.

Purpose of the Study:

• To develop an advanced video generation pipeline for more accurate audio-driven facial movement synthesis.
• To enhance the realism and control of generated portraits in interactive scenarios.

Main Methods:

• A cross-modal Transformer architecture is employed, partitioning facial features into audio-sensitive lower and independent upper regions.
• A novel cross-modal attention module with alignment bias and causal attention manages frame-to-frame facial motion.
• Extended self-attention and self-supervised speech representations address long-term uncertainties and data scarcity.

Main Results:

• The proposed method demonstrates superior control over facial movements and generates high-quality portrait videos.
• Experiments confirm the effectiveness of the audio-to-action mapping and enhanced neural rendering.
• The pipeline significantly improves upon current audio-driven talking face generation techniques.

Conclusions:

• The Transformer-based pipeline offers a robust solution for audio-driven facial animation and interactive video generation.
• This approach advances the state-of-the-art in generating realistic and synchronized talking head videos.