Toward Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Generating realistic 3D body movements from speech is crucial for applications like virtual reality and animation.
• Current methods often struggle with naturalness and coordination due to limitations in modeling inter- and intra-correlations across different body parts (head, body, hands).

Purpose of the Study:

• To propose a novel Hierarchical Implicit Periodicity (HIP) learning approach for audio-inspired 3D gesture generation.
• To address the ill-posed nature of co-speech gesture generation by explicitly modeling intrinsic correlations.

Main Methods:

• Utilizing periodic autoencoders to explore gesture motion phase manifolds, capturing both periodic and non-periodic human movement characteristics.
• Implementing cascaded guidance to model the hierarchical relationships between facial expressions, body gestures, and hand movements.

Main Results:

• The proposed HIP approach demonstrates superior performance compared to state-of-the-art methods in quantitative and qualitative evaluations.
• Generated 3D gestures exhibit improved naturalness and coordination on 3D avatars.

Conclusions:

• The HIP learning approach effectively models multi-modal implicit relationships for realistic co-speech gesture generation.
• This method offers a significant advancement in creating natural and coordinated 3D human movements synchronized with speech.