On-the-Fly Modulation for Balanced Multimodal Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning integrates diverse data types to enhance model performance.
• Current joint training methods often lead to under-optimized unimodal representations due to dominant modalities.

Purpose of the Study:

• To address the issue of imbalanced modality optimization in multimodal learning.
• To develop novel strategies for improving uni-modal representation learning within joint training frameworks.

Main Methods:

• Analysis of under-optimization in feed-forward and back-propagation stages.
• Introduction of On-the-fly Prediction Modulation (OPM) to dynamically drop features of dominant modalities.
• Introduction of On-the-fly Gradient Modulation (OGM) to mitigate gradients of dominant modalities.

Main Results:

• OPM and OGM effectively balance the influence of different modalities during training.
• Demonstrated considerable performance improvements across various multimodal tasks.
• Showcased enhanced performance in both basic and complex multimodal models.

Conclusions:

• The proposed OPM and OGM strategies are effective and flexible solutions for imbalanced multimodal learning.
• These methods offer a simple yet powerful way to improve uni-modal representation quality.
• The findings suggest a new direction for optimizing multimodal learning architectures.