Leveraging hierarchy in multimodal generative models for effective cross-modality inference.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Cross-modality inference (CMI) aims to predict missing data from one sensory modality using data from others.
• Existing single-modality variational autoencoder methods and recent multimodal generative models face challenges in complex CMI tasks.
• Human recognition mechanisms inspire novel approaches to multimodal data representation and inference.

Purpose of the Study:

• To develop a novel hierarchical generative model for unsupervised multimodal representation learning.
• To enable high-quality cross-modality inference for an arbitrary number of modalities.
• To introduce a new benchmark dataset for evaluating CMI in naturalistic, multi-sensory scenarios.

Main Methods:

• Proposed the Nexus model, a hierarchical generative model for unsupervised multimodal representation learning.
• Developed a novel training scheme to ensure robustness to missing modalities during inference.
• Introduced the Multimodal Handwritten Digit (MHD) dataset, integrating image, motion, sound, and label data.

Main Results:

• Nexus generates high-quality, coherent data for missing modalities using any subset of available modalities.
• The hierarchical structure of Nexus is crucial for high-quality sample generation in CMI.
• Nexus demonstrates superior performance compared to state-of-the-art models in cross-modality inference tasks.

Conclusions:

• The Nexus model effectively addresses cross-modality inference challenges through hierarchical representation learning.
• The proposed training scheme enhances model robustness, making it suitable for real-world applications with incomplete data.
• The MHD dataset provides a valuable benchmark for advancing research in multimodal generative modeling and CMI.