Reliability of uncertainty quantification methods for deep learning auto-segmentation in head and neck organs at risk

Area of Science:

• Radiotherapy
• Medical Imaging
• Deep Learning

Background:

• Deep learning auto-segmentation significantly advances radiotherapy contouring but requires quality assurance due to performance variability.
• Manual quality assurance reintroduces variability and diminishes time-saving benefits of auto-segmentation.
• Uncertainty quantification (UQ) is explored to estimate output confidence and improve reliability in auto-segmentation.

Purpose of the Study:

• To compare the reliability of commonly used UQ methods and metrics for radiotherapy auto-segmentation.
• To evaluate the impact of different UQ approaches on segmentation accuracy, confidence calibration, and error localization.

Main Methods:

• Trained a 3D U-Net using the nnU-Net framework for segmenting 19 organs at risk (OAR) in head and neck cancer patients.
• Evaluated three UQ methods: Monte Carlo dropout, deep ensemble modeling, and test-time augmentation.
• Assessed reliability using segmentation accuracy (surface Dice), confidence calibration (ECE-label), and error localization (U-E overlap) on 10 patients.

Main Results:

• Segmentation accuracy was stable across all UQ methods compared to a baseline without UQ.
• UQ methods demonstrated comparable reliability in confidence calibration and error localization.
• The choice of UQ metric significantly influenced reliability, with multi-class predictive entropy outperforming variance and mutual information.

Conclusions:

• The selection of UQ approach substantially impacts the reliability of uncertainty maps in radiotherapy auto-segmentation.
• While UQ methods performed similarly, the specific UQ metric critically affects reliability.
• Careful selection and evaluation of UQ metrics are crucial before clinical application.