Generation in Generation: Fluid Co-Speech Gesture Synthesis With Generative Continuous Quantization

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Conventional co-speech motion generation relies on discrete motion quantization, leading to limited representation accuracy and homogenized motion sequences.
• Existing methods struggle with generating nuanced and contextually appropriate human motion from audio input.

Purpose of the Study:

• To develop a novel explicit generation paradigm for co-speech motion generation that overcomes the limitations of conventional quantization-based methods.
• To enhance the accuracy, smoothness, and diversity of generated human motion representations.
• To improve the generalization capability of motion generation models for real-world applications, particularly in facial style transfer.

Main Methods:

• Introduced a continuous quantization method to derive generative motion units for smoother and more accurate motion representation.
• Proposed a compositional weight generation paradigm for deterministic, explicit motion synthesis, replacing probabilistic sampling.
• Designed a fully audio-aware encoder to extract style features decoupled from content, integrated via Adaptive Instance Normalization for cross-speaker facial style generalization.

Main Results:

• Achieved state-of-the-art performance on two public datasets for co-speech motion generation.
• Demonstrated significantly improved motion representation accuracy and smoothness compared to classical methods.
• Attained an inference speed exceeding 4000 fps on the SHOW dataset, showcasing potential for real-time applications.

Conclusions:

• The proposed generative continuous quantization and compositional weight generation paradigm offers a more effective approach to co-speech motion generation.
• The audio-aware encoder and Adaptive Instance Normalization effectively enhance cross-speaker facial style generalization.
• The method's high inference speed and accuracy position it as a strong candidate for practical, real-time human motion synthesis applications.