Comparing artificial intelligence and physician performance in predicting IDH mutation status in glioma

Area of Science:

• Neuroimaging
• Artificial Intelligence
• Oncology

Background:

• Predicting isocitrate dehydrogenase (IDH) mutations in gliomas via magnetic resonance imaging (MRI) is crucial for treatment planning.
• Artificial intelligence (AI) models offer potential for automated IDH mutation status prediction.

Purpose of the Study:

• To compare the performance of two AI models (GliomaDepth-IDH and GliomaVista-IDH) against physicians in predicting IDH mutations in gliomas.
• To evaluate the generalizability and calibration of AI models on external datasets.

Main Methods:

• Two AI models, ResNet34-based GliomaDepth-IDH and Vision Transformer-based GliomaVista-IDH, were developed and tested.
• AI model performance was compared with 18 physicians (neuroradiologists, neurosurgeons, residents) using AUC and Brier scores.
• Models and physicians were evaluated on the Brain Tumor Segmentation Challenge dataset and a Japanese cohort.

Main Results:

• GliomaVista-IDH achieved a high AUC (0.97) on the initial dataset, outperforming all physician groups.
• On external validation, AI model performance degraded (AUCs 0.75-0.82), with GliomaVista-IDH showing calibration issues (Brier score 0.32).
• High-performing physicians achieved comparable AUC (0.88) with superior calibration (Brier score 0.19), and significant inter-physician variability was observed.

Conclusions:

• AI models can serve as valuable assistants for many physicians in predicting glioma IDH mutations.
• Experienced physicians demonstrate competitive and better-calibrated predictive performance, particularly in complex clinical scenarios.
• Further research is needed to improve AI model robustness and calibration for real-world clinical application.