Harmonized Multimodal Learning with Gaussian Process Latent Variable Models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Multimodal learning seeks relationships between different data types, crucial for applications like cross-modal retrieval.
• Existing Gaussian process latent variable models (GPLVMs) often treat modalities and hyperparameters independently, missing key interdependencies.

Purpose of the Study:

• To address modality heterogeneity in multimodal learning using Gaussian process latent variable models (GPLVMs).
• To develop a novel joint learning scheme, termed Harmonization, for GPLVMs that exploits complementarity between modalities and model components.

Main Methods:

• Proposed a Harmonization mechanism for jointly learning latent representations and kernel hyperparameters in multimodal GPLVMs.
• Derived Harmonization in a model-driven way to enforce agreement between modality-specific GP kernels and latent representation similarity.
• Integrated the Harmonization mechanism into several representative GPLVM-based multimodal learning models.

Main Results:

• Experimental results on four benchmark datasets demonstrated superior performance for the proposed models in cross-modal retrieval tasks.
• The Harmonization method proved effective in discovering semantically consistent latent representations across modalities.
• The novel approach outperformed strong baselines in benchmark cross-modal retrieval evaluations.

Conclusions:

• The Harmonization scheme effectively addresses modality heterogeneity in multimodal GPLVMs.
• Jointly learning latent representations and kernel hyperparameters significantly enhances cross-modal retrieval performance.
• The proposed method advances multimodal learning by fostering semantically coherent representations.