High-Fidelity and Lip-Synced Talking Face Synthesis via Landmark-Based Diffusion Model.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Audio-driven talking face generation is crucial for various industries.
• Existing methods struggle with audio-visual mapping ambiguity and error accumulation.
• Generative adversarial networks (GANs) used in prior work suffer from training instability.

Purpose of the Study:

• To propose a novel landmark-based diffusion model for talking face generation.
• To enable end-to-end optimization for improved lip synchronization.
• To preserve subject appearance details from reference images.

Main Methods:

• Leveraging facial landmarks as intermediate representations for audio-to-landmark motion mapping.
• Introducing TalkFormer, a conditioning module with differentiable cross-attention for motion alignment.
• Employing implicit feature warping to align reference image features with target motion.

Main Results:

• The proposed model synthesizes high-fidelity talking face videos.
• Achieved improved lip synchronization through end-to-end optimization.
• Successfully preserved subject appearance details from reference images.

Conclusions:

• The novel landmark-based diffusion model effectively addresses challenges in talking face generation.
• The TalkFormer module enhances lip synchronization and appearance preservation.
• This approach offers a stable and efficient solution for realistic video synthesis.