MulSMo: Multimodal Stylized Motion Generation by Bidirectional Control Flow

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Generating stylized motion sequences requires balancing content and style.
• Existing methods often suffer from style-content conflicts due to unidirectional information flow.

Purpose of the Study:

• To develop a novel method for stylized motion generation with improved style-content integration.
• To enable flexible multimodal control over motion generation.
• To enhance the performance and dynamics preservation in stylized motion synthesis.

Main Methods:

• Implemented a bidirectional control flow between style and content information.
• Introduced contrastive learning for extending stylized motion generation to text and image modalities.
• Advanced motion diffusion models to motion-aligned temporal latent diffusion using a novel motion VAE (Variational Autoencoder).

Main Results:

• The proposed method significantly outperforms previous approaches across various datasets.
• Demonstrated successful alleviation of style-content collisions and better preservation of style dynamics.
• Enabled effective multimodal signal control for stylized motion generation.

Conclusions:

• The bidirectional control flow effectively integrates style and content in motion generation.
• Multimodal control enhances the flexibility and applicability of stylized motion synthesis.
• The novel motion VAE and diffusion approach achieve state-of-the-art performance.