RoMo: Robust Unsupervised Multimodal Learning With Noisy Pseudo Labels.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• The metaverse and increasing 2D/3D data necessitate cross-modal retrieval.
• Existing methods require costly labeled data, making unsupervised learning desirable.
• Unsupervised cross-modal learning struggles with semantic correlation due to missing labels.

Purpose of the Study:

• To address unsupervised cross-modal learning with noisy pseudo-labels.
• To propose a novel 2D-3D unsupervised multimodal learning framework.
• To improve semantic retrieval across different data modalities.

Main Methods:

• A framework with Self-matching Supervision Mechanism (SSM) for initial discrimination.
• Robust Discriminative Learning (RDL) with Robust Concentrating Learning Loss (RCLL) to handle noisy pseudo-labels.
• Modality-invariance Learning Mechanism (MLM) to create common representations.

Main Results:

• The proposed framework demonstrates effectiveness in unsupervised cross-modal learning.
• It outperforms 14 state-of-the-art methods on four 2D-3D multimodal datasets.
• The method shows robustness against noisy pseudo-labels.

Conclusions:

• The novel framework successfully tackles unsupervised cross-modal learning with noisy pseudo-labels.
• It offers a feasible solution for semantic retrieval in the metaverse.
• The approach enhances discrimination and reduces cross-modal discrepancies.