CLIP-Actor-X: Text-Driven 4D Human Avatar Generation via Cross-Modal Synthesis-Through-Optimization.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Generating realistic 4D human avatars from text prompts is challenging.
• Existing methods often require artist-designed meshes and post-processing for animation and texturing.

Purpose of the Study:

• To develop a novel system, CLIP-Actor-X, for text-driven 4D human avatar generation.
• To create animatable 3D human meshes with motion and texture directly from natural language prompts.

Main Methods:

• A retrieval-augmented generative model with a text-to-motion diffusion model for realistic human motion synthesis.
• A zero-shot neural style optimization module for mesh detailization and texturization.
• Spatio-temporal view augmentation and visibility-aware embedding attention for optimization stability.

Main Results:

• CLIP-Actor-X successfully generates detailed 3D human meshes, motion animations, and textures.
• The generated avatars are animatable, temporally consistent, and pose-agnostic.
• The system aligns well with text prompts without requiring post-processing like re-alignment or rigging.

Conclusions:

• CLIP-Actor-X provides a robust solution for generating high-quality, text-driven 4D human avatars.
• The system demonstrates perceptually plausible and human-recognizable avatars with detailed geometry and texture.
• This approach simplifies the avatar creation pipeline by eliminating the need for manual rigging and post-processing.