Modality-Mix Learning: Promoting Multimodal Learning Through Multilabel Objective

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal fusion integrates diverse data sources for comprehensive understanding.
• Existing multimodal learning methods often suffer from optimization imbalance, leading to incomplete learning of partial modalities.
• Current approaches inadequately address the learning objective's role in balancing modalities.

Purpose of the Study:

• To propose a novel multimodal learning method, modality-mix learning (MM learning), to address the insufficient learning of modalities.
• To promote sufficient learning of each modality by designing a specific multilabel objective.
• To enhance the performance and robustness of multimodal fusion strategies.

Main Methods:

• Developed modality-mix learning (MM learning) that generates modality-mixed samples by combining data from different samples with varied labels.
• Transformed single labels into probability vectors representing multilabel information for training.
• Introduced a bilevel learning scheme: initial standard learning followed by MM learning on selected samples for subexploration modality optimization.

Main Results:

• MM learning effectively forces different objective information to be learned from different modalities, overcoming uniform learning objective limitations.
• The method significantly boosts performance across various fusion strategies and methods on diversified multimodal datasets.
• Demonstrated improved robustness of multimodal networks through the proposed MM learning approach.

Conclusions:

• Modality-mix learning provides an effective solution to the challenge of insufficient modality learning in multimodal AI.
• The proposed method enhances the discriminative power of multimodal networks by optimizing individual modality learning.
• MM learning offers a promising direction for advancing multimodal fusion and robust AI systems.