Audio2Gestures: Generating Diverse Gestures From Audio.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Generating co-speech gestures from audio is complex due to the inherent one-to-many relationship between speech and motion.
• Traditional models often predict average motions, leading to less diverse and engaging gestures.

Purpose of the Study:

• To develop a novel approach for co-speech gesture generation that explicitly models the one-to-many audio-to-motion mapping.
• To enhance the diversity and realism of generated gestures compared to existing methods.

Main Methods:

• Proposed a Variational Autoencoder (VAE) framework that splits cross-modal latent codes into shared (audio-correlated) and motion-specific (diverse) components.
• Introduced specialized training losses, including relaxed motion loss, bicycle constraint, and diversity loss, to address training complexities.
• Validated the approach on 3D and 2D motion datasets, incorporating structured losses (e.g., STFT) for improved motion evaluation.

Main Results:

• The proposed method significantly outperforms state-of-the-art approaches in generating more realistic and diverse co-speech gestures, both quantitatively and qualitatively.
• Demonstrated compatibility with various backbones like RNNs, Transformers, and Discrete Cosine Transform (DCT) modeling.
• Showcased the ability to generate motion sequences with user-specified motion clips.

Conclusions:

• The novel latent code splitting strategy effectively captures diverse gestural information independent of audio.
• The developed training strategies and evaluation metrics lead to superior motion dynamics and nuanced details in generated gestures.
• The method offers a flexible and powerful solution for realistic and controllable co-speech gesture generation.