STAR: Skeleton-aware Text-based 4D Avatar Generation with in-Network Motion Retargeting.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Current methods for 4D avatar generation from text rely on text-to-image models, which face challenges with domain gaps and view-consistency.
• Post-hoc animation techniques often result in translation artifacts and misalignment due to differences in avatar geometry and motion.

Purpose of the Study:

• To develop a robust framework for generating high-quality, animated 4D avatars directly from text descriptions.
• To address the limitations of existing optimization-by-animation paradigms in synthesizing and animating 3D avatars.

Main Methods:

• Proposed Skeleton-aware Text-based 4D Avatar generation with in-network motion Retargeting (STAR).
• Incorporated skeleton-aware motion retargeting to correct mismatched source motions based on geometry and skeleton differences.
• Utilized a hybrid Score Distillation Sampling (SDS) module with skeleton-conditioned text-to-image (T2I) and text-to-video (T2V) priors for multiview and frame-consistent supervision.

Main Results:

• STAR enables end-to-end optimization of avatar geometry, texture, and motion.
• Demonstrated the synthesis of high-quality 4D avatars with vivid animations that align with text descriptions.
• Quantitative and qualitative experiments confirmed the effectiveness of the proposed method.

Conclusions:

• STAR effectively generates high-fidelity 4D avatars with accurate and natural animations.
• The proposed approach overcomes key limitations in previous text-to-4D avatar generation techniques.
• Ablation studies validated the contribution of individual components within the STAR framework.