Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Oncology
• Neurosurgery and Radiology

Background:

• Deep learning algorithms are increasingly used for brain tumor segmentation.
• Current evaluation practices often rely on quantitative metrics, with limited inclusion of clinical expert assessment.
• Assessing segmentation quality is crucial for effective cancer treatment planning.

Purpose of the Study:

• To survey current practices in evaluating deep learning segmentation algorithms for brain tumors.
• To investigate expert perception of segmentation quality and its correlation with quantitative metrics.
• To highlight the gap between automated metrics and clinical judgment in cancer research.

Main Methods:

• Literature survey of 180 articles on brain tumor segmentation algorithms.
• Collected quality ratings from medical professionals on 60 brain tumor segmentation cases.
• Analyzed interrater agreement and correlations between expert ratings and standard metrics (Dice, Hausdorff distance).

Main Results:

• Dice score, sensitivity, and Hausdorff distance are common metrics, but expert evaluation is rare (2.8% of articles).
• Low interrater agreement (Krippendorff α = 0.34) among experts in quality perception.
• Weak correlations found between expert ratings and quantitative metrics (Dice: 0.23, Hausdorff: 0.51).

Conclusions:

• Expert quality ratings for brain tumor segmentation are highly variable due to ambiguous boundaries and individual differences.
• Existing quantitative metrics inadequately capture the clinical perception of segmentation quality.
• There is a critical need to develop evaluation methods that better align with clinical relevance for cancer patients.