Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Generating realistic 3D human motion synchronized with speech is complex.
• Existing models struggle with multiple inputs (speech, identity, emotion) and coordinating facial/body movements.

Purpose of the Study:

• To propose a novel framework, Combo, for harmonious co-speech 3D human motion generation.
• To enable efficient and customizable adaptation of generated motions for identity and emotion.

Main Methods:

• Introduced a pre-training strategy on neutral data, followed by fine-tuning with a novel X-Adapter for efficient adaptation.
• Developed DU-Trans, a transformer architecture that branches for facial and body motion, then merges for coordinated prediction.

Main Results:

• Combo effectively generates high-quality, holistic 3D human motions synchronized with speech.
• The framework demonstrates efficiency in adapting motions for varying identity and emotion.

Conclusions:

• Combo addresses the MIMO challenge in motion generation.
• The proposed methods offer a robust and adaptable solution for co-speech motion synthesis.