Multimodal Weibull Variational Autoencoder for Jointly Modeling Image-Text Data.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Traditional black-box models struggle with interpretable multilayer structures in multimodal representation learning.
• Visualizing connections between modalities at multiple semantic levels is challenging.
• Deep topic models offer a foundation for extracting latent representations.

Purpose of the Study:

• To develop a novel method for extracting interpretable multimodal latent representations.
• To visualize hierarchical semantic relationships between different modalities.
• To improve the efficiency and scalability of multimodal learning.

Main Methods:

• Developed a multimodal Poisson gamma belief network (mPGBN) with sparse connections between modality-specific hidden layers.
• Constructed a Weibull-based variational inference network (encoder) to replace slow Gibbs sampling.
• Combined the encoder with the mPGBN (decoder) to create the multimodal Weibull variational autoencoder (MWVAE).

Main Results:

• MWVAE successfully extracts expressive multimodal latent representations from bimodal image-text data.
• The model demonstrates effectiveness in downstream tasks such as missing modality imputation and multimodal retrieval.
• Both MWVAE and its supervised variant (sMWVAE) achieve state-of-the-art performance on multimodal benchmarks.

Conclusions:

• The proposed MWVAE offers an efficient and interpretable approach to multimodal representation learning.
• The Weibull-based variational inference significantly speeds up prediction for large-scale datasets.
• MWVAE provides a robust framework for handling complex relationships within multimodal data.