Multimodal learning rebalanced: Negative correlation ensembles for improved performance.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Multimodal learning integrates diverse data but faces challenges with modality imbalance, where dominant modalities overshadow others.
• Existing solutions often improve non-dominant modalities at the expense of dominant ones, hindering overall performance.

Purpose of the Study:

• To address the modality imbalance problem in multimodal learning.
• To develop a method that fully leverages information from all modalities, avoiding performance degradation.

Main Methods:

• Proposed a novel approach treating each modality as a basic classifier within an ensemble learning framework.
• Introduced negative correlation learning to promote information diversity across modalities.
• Validated the method using late fusion techniques across multiple datasets and tasks.

Main Results:

• The proposed method demonstrated superior performance compared to existing approaches.
• Achieved significant improvements in accuracy across various tasks and datasets.
• Effectively balanced the utilization of information from all modalities.

Conclusions:

• The ensemble learning perspective effectively resolves the modality imbalance problem in multimodal learning.
• Negative correlation learning enhances the diversity and robustness of multimodal models.
• The proposed method offers a promising direction for developing more effective multimodal AI systems.