Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI

Area of Science:

• Neuroimaging
• Artificial Intelligence
• Medical Diagnostics

Background:

• Accurate glioma segmentation in multimodal MRI is crucial for neurosurgical interventions.
• Deep learning models offer automated lesion segmentation but often lack transparency ('black box' problem).
• Clinical adoption of AI in neurosurgery is hindered by the inability to understand model predictions.

Purpose of the Study:

• To develop a hybrid deep learning model for accurate and robust glioma segmentation in brain MRI.
• To introduce an explainability technique (TransXAI) that provides surgeon-understandable heatmaps without altering model architecture or accuracy.
• To enhance the transparency and clinical trust in AI-driven neuroimaging analysis.

Main Methods:

• Proposed a novel hybrid model combining Vision Transformers and Convolutional Neural Networks (CNNs).
• Implemented a post-hoc explanation technique to generate visual interpretations (heatmaps) of model predictions.
• Utilized multimodal brain MRI volumes for glioma segmentation and analysis.

Main Results:

• TransXAI achieved competitive performance in segmenting gliomas from brain MRI scans.
• The method generated explainable saliency maps, aiding in understanding deep network predictions.
• Visualization maps revealed information flow within the encoder-decoder network and modality contributions.

Conclusions:

• TransXAI provides accurate glioma segmentation with enhanced transparency through explainable heatmaps.
• The explainability feature can increase medical professionals' trust in deep learning systems for clinical use.
• The approach facilitates the integration of AI tools in neurosurgical procedures by demystifying AI decision-making.