Modality Equilibrium Matters: Minor-Modality-Aware Adaptive Alternating for Cross-Modal Memory Enhancement.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal fusion faces challenges with modality imbalance, where dominant data types overshadow weaker ones, leading to biased learning.
• This imbalance is particularly problematic in incomplete modality scenarios, hindering optimal fusion performance.

Purpose of the Study:

• To introduce a novel Equilibrium Deviation Metric (EDM) for quantifying modality imbalance in multimodal fusion.
• To investigate the impact of modality optimization order on achieving equilibrium and improving fusion performance.
• To develop a robust framework addressing modality imbalance and enhancing multimodal learning.

Main Methods:

• Developed the Equilibrium Deviation Metric (EDM) to quantify modality imbalance.
• Proposed an EDM-ranked weak-to-strong modality optimization schedule.
• Designed an alternating strategy with a modality-mapping layer and memory module for dynamic prioritization and feature alignment.

Main Results:

• Demonstrated theoretically and empirically that modality optimization order significantly impacts equilibrium.
• The EDM-ranked weak-to-strong schedule provides the tightest convergence bound.
• Achieved new state-of-the-art results on four benchmarks, including CREMA-D and Kinetics-400, with significant performance gains.
• Framework showed robustness under missing-modality conditions.

Conclusions:

• Modality scheduling is crucial for mitigating imbalance in multimodal fusion.
• The proposed framework offers a principled approach to multimodal learning, outperforming conventional joint training.
• The EDM provides a valuable tool for analyzing and improving multimodal fusion systems.