Multimodal Variational Autoencoder: A Barycentric View.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Real-world phenomena involve multiple signal modalities (e.g., vision, sound).
• Multimodal representation learning using variational autoencoders (VAEs) is gaining traction, especially for handling missing modalities.
• Existing multimodal VAEs often rely on expert aggregation methods like Product of Experts (PoE) or Mixture of Experts (MoE).

Purpose of the Study:

• To propose a new theoretical formulation for multimodal VAEs based on the concept of barycenters.
• To demonstrate that existing PoE and MoE methods are specific instances of barycenters.
• To introduce a more flexible barycenter approach using different divergence measures, particularly the Wasserstein distance.

Main Methods:

• Developed a generic theoretical formulation for multimodal VAEs using barycenters.
• Showed that Product of Experts (PoE) and Mixture of Experts (MoE) are special cases of barycenters derived from KL divergence.
• Introduced and explored the Wasserstein barycenter, utilizing the 2-Wasserstein distance for improved representation learning.

Main Results:

• The proposed barycenter formulation extends existing methods by allowing for more flexible choices of divergence.
• The Wasserstein barycenter effectively captures both modality-invariant and modality-specific representations by preserving the geometry of unimodal distributions.
• Empirical evaluations on three multimodal benchmarks confirmed the superior performance of the proposed method.

Conclusions:

• The barycenter framework offers a more generalized and flexible approach to multimodal VAEs compared to expert-based methods.
• The Wasserstein barycenter provides enhanced representation learning by better preserving distributional geometry.
• The proposed method demonstrates significant effectiveness in multimodal representation learning tasks, particularly with missing modalities.