Multi-Modal Latent Diffusion.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal datasets are increasingly common in AI applications.
• Multimodal Variational Autoencoders (MVAEs) aim to learn joint representations across different data types.
• Existing MVAEs face a trade-off between generation quality and cross-modal coherence.

Purpose of the Study:

• To address the limitations of current multimodal Variational Autoencoders.
• To develop a novel method for improved multimodal generative modeling.
• To enhance both the quality and coherence of generated multimodal data.

Main Methods:

• Utilized independently trained unimodal deterministic autoencoders.
• Concatenated individual latent variables into a shared latent space.
• Employed a masked diffusion model for generative tasks.
• Introduced a multi-time training strategy for conditional score networks in multimodal diffusion.

Main Results:

• The proposed method significantly outperforms existing approaches.
• Demonstrated substantial improvements in generation quality.
• Achieved superior generative coherence across different modalities.
• Validated through an extensive experimental campaign.

Conclusions:

• The novel approach effectively resolves the coherence-quality tradeoff in MVAEs.
• The method offers a promising direction for advanced multimodal generative modeling.
• This work sets a new benchmark for generating high-quality, coherent multimodal data.