DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Text-to-3D avatar generation leverages 2D diffusion models and score distillation sampling (SDS).
• Existing methods struggle with generating high-quality, animatable 3D avatars.
• Challenges include view consistency, pose accuracy, and realistic animation.

Purpose of the Study:

• Introduce DreamWaltz-G, a novel framework for text-driven animatable 3D avatar generation.
• Enhance the quality and expressiveness of generated 3D avatars.
• Enable real-time rendering, stable optimization, and diverse animation capabilities.

Main Methods:

• Skeleton-guided Score Distillation integrates 3D human template skeletons into 2D diffusion models for improved SDS supervision.
• Hybrid 3D Gaussian Avatar representation combines neural implicit fields and 3D meshes for efficient rendering and animation.
• Utilizes pretrained 2D diffusion models and score distillation sampling (SDS).

Main Results:

• DreamWaltz-G effectively generates high-quality 3D avatars, mitigating common issues like multiple faces and blurring.
• The framework achieves superior visual quality and animation expressiveness compared to existing methods.
• Demonstrates successful applications in human video reenactment and multi-subject scene composition.

Conclusions:

• DreamWaltz-G offers a robust solution for generating and animating high-fidelity 3D avatars from text.
• The proposed methods significantly advance the state-of-the-art in text-to-3D avatar creation.
• The framework's versatility supports various downstream applications in computer graphics and AI.