A Real-World Evaluation of Failure Detection for Liver CT Segmentation

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Deep learning models in clinical imaging face distribution shifts.
• Current out-of-distribution (OOD) detection methods are often evaluated on limited research datasets, raising concerns about real-world clinical applicability.
• Reliable identification of segmentation failures in clinical practice remains a challenge.

Purpose of the Study:

• To evaluate the effectiveness of existing OOD detection methods in identifying segmentation failures in a deployed liver CT segmentation model.
• To introduce and assess a novel surface-based OOD detection method, Pairwise Surface DSC.
• To determine the clinical feasibility of automated failure detection in liver CT segmentation.

Main Methods:

• Six OOD detection methods were evaluated on a 3D nnU-Net liver CT segmentation model.
• Data included internal (400 patients) and external (100 patients from ~70 sites globally) cohorts.
• Performance metrics included sensitivity, AUROC, and balanced accuracy, with thresholds set using the Youden J statistic on an independent cohort (400 patients).
• Statistical significance was assessed using McNemar tests and stratified bootstraps with Benjamini-Hochberg correction.

Main Results:

• Pairwise Surface DSC outperformed other methods, achieving perfect sensitivity (1.00).
• It demonstrated near-perfect AUROCs (0.97 internal; 1.00 external) and the highest balanced accuracies (0.94 internal; 0.88 external).
• Statistical analysis confirmed the significant superiority of Pairwise Surface DSC (p < 0.001).

Conclusions:

• Automated failure detection for liver CT segmentation is clinically feasible.
• Pairwise Surface DSC is a highly effective and promising method for identifying OOD data in clinical imaging.
• The findings support the potential deployment of Pairwise Surface DSC for enhancing the reliability of AI in medical imaging.