PSL: Prototype swapping learning for modality imbalance.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning (MML) leverages diverse data types for enhanced task performance.
• Existing MML methods struggle with modality imbalance, where one modality dominates, degrading overall performance.
• Current solutions for modality imbalance are often complex and require explicit imbalance rate estimation.

Purpose of the Study:

• To introduce a novel framework, Prototype Swapping Learning (PSL), to address modality imbalance in MML.
• To develop a self-adaptive method that does not require explicit imbalance rate calculation.
• To improve the performance of existing MML methods by seamlessly integrating PSL.

Main Methods:

• Developed the Prototype Swapping Learning (PSL) framework for MML.
• PSL utilizes a mutual learning approach by swapping prototypes between modalities.
• The framework promotes slow-learning modalities using dominant ones and vice-versa, adapting automatically.

Main Results:

• PSL effectively remedies modality imbalance in a self-adaptive manner.
• The proposed method significantly improves performance on multiple benchmark datasets.
• Experiments demonstrate the superiority of PSL over state-of-the-art MML approaches.

Conclusions:

• Prototype Swapping Learning (PSL) is an effective and novel approach to mitigate modality imbalance in multimodal learning.
• PSL offers a simple yet powerful self-adaptive solution, enhancing existing MML techniques.
• The framework's seamless integration and demonstrated effectiveness make it a valuable contribution to the field of MML.