Toward Fine-Grained Talking Face Generation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Talking face generation synthesizes lip-synchronized videos from portraits and audio.
• Existing methods struggle with vivid facial expressions, smooth transitions, and portrait detail preservation.
• Previous approaches often result in low-fidelity videos with unnatural facial muscle movements.

Purpose of the Study:

• To propose a novel model, Fine-gRained mOtioN moDel (FROND), for high-fidelity talking face generation.
• To address limitations in capturing facial expressions, ensuring temporal smoothness, and maintaining portrait details.
• To improve the realism and quality of synthesized talking face videos.

Main Methods:

• A two-stream encoder captures local facial keypoint movements and global motion context.
• A motion estimation module predicts audio-driven movements, learning continuous trajectories for smooth temporal and spatial motion.
• An implicit neural network-based decoder recovers high-frequency details for high-fidelity video synthesis.

Main Results:

• FROND refines facial keypoint trajectories into a continuous dense motion field.
• The model effectively fuses local and global motion information for smooth movements.
• Quantitative and qualitative evaluations demonstrate superior performance over state-of-the-art baselines.

Conclusions:

• FROND significantly enhances the quality of talking face generation.
• The proposed method achieves state-of-the-art results in synthesizing realistic and detailed talking face videos.
• FROND offers a robust solution for generating fine-grained facial animations.