SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting

Area of Science:

• Computer Vision and Graphics
• Artificial Intelligence
• Multimedia Synthesis

Background:

• Creating realistic, speech-driven talking head videos is challenging due to the difficulty in synchronizing subject identity, lip movements, facial expressions, and head poses.
• Lack of synchronization leads to fundamental flaws and unrealistic results in talking head synthesis.

Purpose of the Study:

• To introduce SyncTalk++, a novel framework designed to address the critical synchronization challenges in realistic talking head video generation.
• To improve the realism, stability, and rendering speed of speech-driven talking head synthesis.

Main Methods:

• Employed a Dynamic Portrait Renderer with Gaussian Splatting for consistent subject identity preservation.
• Introduced a Face-Sync Controller utilizing a 3D facial blendshape model for accurate lip-sync and facial expression reconstruction.
• Developed a Head-Sync Stabilizer for optimizing head pose stability and incorporated an Expression Generator and Torso Restorer for robustness to out-of-distribution audio.

Main Results:

• SyncTalk++ achieves high synchronization across lip movements, facial expressions, and head poses, significantly improving realism.
• The system demonstrates robustness to out-of-distribution audio and maintains visual consistency and continuity across frames.
• Achieved rendering speeds of up to 101 frames per second, outperforming state-of-the-art methods in both synchronization and realism.

Conclusions:

• SyncTalk++ effectively overcomes the "devil" of synchronization in talking head synthesis, delivering highly realistic and stable results.
• The proposed methods enhance visual quality, rendering speed, and robustness, setting a new benchmark for speech-driven talking head generation.