Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Multi-modal learning integrates diverse data types but suffers from modality imbalance, where dominant modalities overshadow others.
• Existing methods use global parameter updates, failing to account for individual parameter importance.

Purpose of the Study:

• To address modality imbalance in multi-modal learning by proposing a novel element-wise joint optimization strategy.
• To improve the overall effectiveness of multi-modal models by ensuring balanced optimization across all modalities.

Main Methods:

• Proposed Adaptively Mask Subnetworks Considering Modal Significance (AMSS), an importance sampling-based element-wise joint optimization method.
• Utilized mutual information rates to determine modal significance and adaptive sampling for parameter updates.
• Introduced AMSS+, an enhanced version employing unbiased estimation for improved subnetwork strategies.

Main Results:

• Demonstrated the effectiveness of importance sampling over uniform sampling and global-wise updating.
• AMSS and AMSS+ significantly outperformed existing methods in rebalancing multi-modal learning.
• Convergence analysis confirmed the reliability of the AMSS strategy.

Conclusions:

• AMSS provides a robust solution to modality imbalance in multi-modal learning.
• Element-wise optimization with adaptive sampling is crucial for achieving joint optima.
• The proposed methods enhance model performance by effectively balancing diverse data modalities.