Analysis of annotation requirements for training uncertainty-aware image segmentation models.

Area of Science:

• Medical image analysis
• Computer vision
• Machine learning

Background:

• Uncertainty quantification is crucial for image segmentation models in expert-variability scenarios.
• A key question is whether multi-annotator training is needed for robust uncertainty estimates.

Purpose of the Study:

• To investigate if single-annotator training suffices for robust uncertainty quantification in image segmentation.
• To compare single- versus multi-annotator training strategies across diverse datasets and models.

Main Methods:

• Utilized nine public datasets with the nnU-Net framework.
• Extended nnU-Net with Ensemble, Bayesian, Probabilistic, and Hierarchical Probabilistic models.
• Assessed uncertainty using probability maps and disagreement-as-class, measured by A-Dice, GED, and Dice.

Main Results:

• Ensemble probability map methods performed consistently well in both single- and multi-annotation settings.
• Disagreement-as-class methods showed significant advantages with multi-annotation for capturing inter-expert variability.
• Analytical arguments supported empirical findings.

Conclusions:

• Multiple annotations per case may not be essential for training effective uncertainty-aware segmentation models.
• Findings suggest reduced annotation costs and scalable development of reliable systems.
• Ensemble probability map approaches offer a practical solution for uncertainty quantification.