Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts.

Area of Science:

• Computer Vision and Graphics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Current audio-driven human video synthesis methods often struggle with precise lip synchronization, expressive body gestures, and consistent visual quality.
• Generating photorealistic and temporally coherent 3D talking videos remains a significant challenge in computer graphics and AI.

Purpose of the Study:

• To introduce Stereo-Talker, a novel one-shot audio-driven system for generating high-fidelity 3D talking videos.
• To achieve precise lip synchronization, expressive body gestures, and continuous viewpoint control in synthesized videos.
• To enhance motion diversity and video generation stability through advanced AI techniques.

Main Methods:

• A two-stage approach: first, mapping audio to high-fidelity motion sequences using large language model (LLM) priors and semantic audio features.
• Second, improving diffusion-based video generation with a prior-guided Mixture-of-Experts (MoE) mechanism (view-guided and mask-guided).
• Development of a mask prediction module for enhanced mask stability and accuracy during inference.

Main Results:

• Successful generation of 3D talking videos with accurate lip synchronization and expressive, temporally consistent body gestures.
• Demonstrated enhanced motion quality and generation stability through LLM integration and MoE mechanisms.
• Creation of a comprehensive human video dataset with 2,203 identities for improved model generalization.

Conclusions:

• Stereo-Talker represents a significant advancement in audio-driven 3D human video synthesis.
• The system effectively combines LLM priors and MoE diffusion models for high-quality, controllable video generation.
• The released dataset and models will facilitate further research in realistic human video synthesis.