Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multimodal learning leverages multiple data sources for enhanced task performance.
• Robustness to missing or corrupted data is crucial for real-world multimodal systems.
• Existing multimodal networks often suffer significant performance degradation when modalities are absent.

Purpose of the Study:

• To develop a parameter-efficient adaptation procedure for pretrained multimodal networks.
• To enhance the robustness of multimodal systems to missing modalities at test time.
• To enable multimodal networks to compensate for absent data through feature modulation.

Main Methods:

• Proposing a simple and parameter-efficient adaptation procedure for pretrained multimodal networks.
• Exploiting modulation of intermediate features to compensate for missing modalities.
• Evaluating the method across various modality combinations and downstream tasks.

Main Results:

• The proposed adaptation method partially bridges the performance gap caused by missing modalities.
• The method outperforms independent, dedicated networks for available modality combinations in some cases.
• The adaptation requires a minimal number of parameters (less than 1% of total parameters).

Conclusions:

• The proposed method offers a versatile and efficient solution for robust multimodal learning.
• It demonstrates significant improvements in handling missing modalities across diverse tasks and datasets.
• This approach enhances the practical applicability of multimodal systems in scenarios with incomplete data.