What are you looking at? Modality contribution in multimodal medical deep learning

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Deep Learning

Background:

• Multimodal data analysis is crucial in medicine due to high-dimensional patient information.
• Deep neural networks (DNNs) are increasingly used for multimodal data analysis.
• Understanding how DNNs process information from individual data sources remains a challenge.

Purpose of the Study:

• To develop and apply a model- and performance-agnostic method for quantifying modality contributions in multimodal deep learning.
• To investigate the detailed information processing of individual sources within multimodal models.
• To enhance the interpretability of deep learning models in multimodal medical research.

Main Methods:

• Implemented an occlusion-based modality contribution method.
• Quantitatively measured the importance of each data modality for model task performance.
• Applied the method to three distinct multimodal medical datasets for validation.

Main Results:

• Identified instances where deep learning networks exhibit modality preferences, leading to unimodal collapses.
• Revealed inherent imbalances within certain multimodal datasets.
• Provided fine-grained quantitative and visual attribute importance assessments for each modality.

Conclusions:

• The developed metric offers valuable insights for advancing multimodal model development and dataset creation.
• This interpretability approach facilitates the integration of multimodal artificial intelligence (AI) into clinical practice.
• The code for the modality contribution method is publicly available to support research.