Multimodal Similarity Gaussian Process Latent Variable Model.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Vision

Background:

• Real-world data often comprises multiple modalities with shared semantics but differing representations.
• Existing methods struggle with the complexity and divergence of multimodal data due to rigid, parameterized mappings.
• A need exists for flexible models that can capture intricate relationships within and across modalities.

Purpose of the Study:

• To develop advanced Gaussian Process Latent Variable Models (GPLVM) for effective multimodal data representation.
• To introduce non-parametric mapping functions that adapt to content divergence and semantic complexity.
• To improve cross-modal retrieval and multimodal classification performance.

Main Methods:

• Building upon Gaussian Process Latent Variable Models (GPLVM).
• Proposing four novel models: m-SimGP, m-DSimGP, m-RSimGP, and m-DRSimGP.
• Utilizing gradient descent for scalable optimization of objective functions.

Main Results:

• Demonstrated superior performance on five real-world datasets for cross-modal retrieval and multimodal classification.
• Successfully learned non-linear correlations and generated comparable low-dimensional representations for heterogeneous modalities.
• The proposed m-DRSimGP model effectively combines distance and semantic preservation.

Conclusions:

• The developed GPLVM-based approaches offer a flexible and powerful solution for multimodal data analysis.
• These methods significantly advance the state-of-the-art in cross-modal understanding and classification.
• The models provide a robust framework for discovering latent structures in complex, heterogeneous datasets.