Incomplete Modalities Restoration via Hierarchical Adaptation for Robust Multimodal Segmentation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Multimodal semantic segmentation integrates diverse data sources for enhanced understanding.
• Missing modalities, caused by sensor failures or data errors, significantly degrade segmentation performance.
• Current methods require extensive computational resources for specialized models per missing scenario.

Purpose of the Study:

• To propose a Hierarchical Adaptation framework to Restore Missing Modalities for Multimodal segmentation (HARM3).
• To enable direct application of frozen pre-trained multimodal models to missing-modality tasks with minimal parameter updates.
• To enhance model robustness and adaptability in scenarios with prevalent missing modalities.

Main Methods:

• HARM3 utilizes a text-instructed missing modality prompt module to generate prompts for missing data.
• This module leverages available modalities and textual instructions to learn multimodal semantic knowledge.
• The framework incorporates adaptive perturbation training and an affine modality adapter for improved robustness.

Main Results:

• HARM3 effectively restores missing modalities in multimodal semantic segmentation.
• The framework demonstrates minimal parameter updates for adapting pre-trained models.
• Experiments confirm HARM3's effectiveness and robustness across various missing modality scenarios.

Conclusions:

• HARM3 offers an efficient solution for missing-modality semantic segmentation.
• The proposed framework facilitates knowledge transfer from high-resource to low-resource domains.
• HARM3 significantly advances the applicability of multimodal models in real-world scenarios.