Multimodal Contrastive Prototype Learning for Resilient Brain Tumor Segmentation With Missing Modalities.

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Computational Neuroscience

Background:

• Multimodal fusion is crucial for comprehensive brain tumor diagnosis but struggles with missing data.
• Existing models face challenges in simultaneously extracting modality-specific and modality-shared features.

Purpose of the Study:

• To propose a two-stage dual-view prototype learning framework for robust brain tumor diagnosis.
• To address the challenge of missing modalities in multimodal medical imaging.
• To simultaneously extract modality-specific and class-specific features.

Main Methods:

• A two-stage framework utilizing Transformer decoders for modality and class prototype learning.
• Incorporation of masked autoencoders for shared feature generation and masked modality strategy for handling missing data.
• Application of modality-view and class-view contrastive learning to improve prototype representation.

Main Results:

• Demonstrated superior performance on BraTS2020 and BraTS2018 datasets, especially with missing modalities.
• Achieved significant Dice Similarity Coefficient (DSC) improvements: 5.9% for Enhancing Tumor (ET), 0.5% for Tumor Core (TC), and 0.2% for Whole Tumor (WT) on BraTS2020.
• Showcased clinically significant gains in T1C modality missing scenarios: 9.5% for ET and 1.8% for TC.

Conclusions:

• The proposed framework effectively handles missing modalities in brain tumor diagnosis.
• Dual-view prototype learning enhances the extraction of critical features for accurate segmentation.
• The model offers a promising solution for improving diagnostic accuracy in challenging clinical scenarios.