Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results

Area of Science:

• Medical Image Analysis
• Artificial Intelligence in Medicine
• Computational Pathology

Background:

• Deep learning (DL) dominates medical image segmentation but faces challenges in reliability and clinical use.
• Annotation variability, model calibration, and uncertainty estimation are key hurdles for DL models.
• Leveraging inter-annotator variability is essential for robust evaluation of segmentation models.

Purpose of the Study:

• To introduce the CURVAS challenge, focusing on multi-rater annotation for robust medical image segmentation.
• To evaluate DL models' calibration and uncertainty estimation capabilities in segmentation tasks.
• To assess the impact of multi-annotator ground truth on model performance and reliability.

Main Methods:

• Seven teams developed DL models for multi-organ segmentation.
• Models were evaluated using Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS).
• Consensus and dissensus ground truth derived from multiple annotators were used for evaluation.

Main Results:

• Well-calibrated models showed a strong correlation with high-quality segmentation results.
• Models trained on diverse datasets and pre-trained knowledge demonstrated increased robustness.
• Top-performing models achieved high DSC and accurate uncertainty estimates.

Conclusions:

• Multi-annotator ground truth is vital for developing trustworthy DL segmentation models.
• Thorough calibration assessment and uncertainty-aware evaluations are necessary for clinical reliability.
• Addressing annotation variability through multi-rater approaches enhances DL model performance and trustworthiness.