I²MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multimodal Learning

Background:

• Modality fusion is crucial for integrating diverse data sources in multimodal learning.
• Existing fusion methods struggle with heterogeneous interactions and lack interpretability.
• Uncovering complex multimodal interactions is essential for advanced AI applications.

Purpose of the Study:

• To propose an end-to-end framework, Interpretable Multimodal Interaction-aware Mixture of Experts (I²MoE), to address limitations in current modality fusion techniques.
• To enhance modality fusion by explicitly modeling diverse multimodal interactions and providing both local and global interpretation.
• To develop a flexible and interpretable solution for multimodal learning tasks.

Main Methods:

• I²MoE employs an end-to-end Mixture of Experts (MoE) framework.
• Utilizes distinct interaction experts with weakly supervised losses to learn data-driven multimodal interactions.
• Incorporates a reweighting model to assign importance scores for expert outputs, enabling sample-level and dataset-level interpretation.

Main Results:

• I²MoE demonstrates flexibility by integrating with various fusion techniques.
• Consistently improves task performance across medical and general multimodal datasets.
• Provides robust interpretation capabilities for multimodal data analysis.

Conclusions:

• I²MoE offers a significant advancement in interpretable modality fusion.
• The framework enhances understanding of multimodal interactions in diverse applications.
• I²MoE provides a valuable tool for researchers and practitioners in multimodal AI.