Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation.

Area of Science:

• Computer Graphics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Singing-driven 3D head animation is crucial for virtual avatars but current speech-driven methods produce limited results.
• Existing techniques struggle with emotional flatness and semantic inconsistency, failing to meet singing animation's unique needs.

Purpose of the Study:

• To introduce Think2Sing, a novel diffusion-based framework for generating semantically consistent and temporally coherent 3D head animations from singing.
• To address the limitations of existing methods by integrating large language models and a new motion subtitle representation.

Main Methods:

• Developed a unified diffusion-based framework (Think2Sing) leveraging pretrained large language models.
• Introduced 'motion subtitles': structured, time-aligned representations generated via a Singing Chain-of-Thought process with acoustic-guided retrieval.
• Formulated animation as motion intensity prediction over key facial regions for fine-grained control and expressive modeling.

Main Results:

• Think2Sing significantly outperforms state-of-the-art methods in realism, expressiveness, and emotional fidelity.
• The framework demonstrates semantically grounded and expressive motion learning capabilities.
• Enabled flexible subtitle-conditioned editing for precise, user-controllable animation synthesis.

Conclusions:

• Think2Sing offers a robust solution for high-quality singing-driven 3D head animation.
• The proposed motion subtitle representation and diffusion framework advance the state-of-the-art in expressive character animation.
• The newly constructed multimodal dataset facilitates future research in this domain.