Speech-Driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Speech-driven gesture generation is crucial for realistic virtual humans.
• Existing methods struggle with complex multimodal inputs and limited expressiveness.
• Accurate feature extraction from speech remains a significant challenge.

Purpose of the Study:

• To introduce Persona-Gestor, an end-to-end generative model for personalized 3D gesture synthesis.
• To overcome limitations of traditional methods by relying solely on raw speech audio.
• To enhance the naturalness, expressiveness, and synchronization of generated gestures.

Main Methods:

• Developed a novel fuzzy feature extractor using fuzzy inference to capture implicit speech features.
• Employed a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture (DiTs-based).
• Integrated fuzzy features into the AdaLN transformer for unified latent representation and gesture generation.

Main Results:

• Persona-Gestor demonstrated superior performance over state-of-the-art methods on Trinity, ZEGGS, and BEAT datasets.
• Achieved high levels of gesture-speech synchronization and naturalness.
• Validated through extensive subjective and objective evaluations.

Conclusions:

• Persona-Gestor sets a new benchmark for speech-driven gesture synthesis.
• The model significantly improves usability and generalization capabilities in virtual human technology.
• Paves the way for more immersive and interactive virtual human experiences.