Benchmarking robustness of automated CT pancreas segmentation: achieving human-level reliability through...

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Computational Anatomy

Background:

• Deep learning models for pancreas segmentation in CT scans have advanced rapidly.
• Current evaluation metrics (e.g., Dice, surface metrics) do not fully capture model robustness, defined as consistent human-level performance across diverse cases.
• Segmentation robustness is critical for clinical applications like early detection and quantitative biomarker analysis.

Purpose of the Study:

• To systematically evaluate the robustness of deep learning models for pancreas segmentation compared to human readers.
• To investigate the effectiveness of an active learning strategy for improving segmentation reliability.

Main Methods:

• Retrospective analysis of 903 CT scans, with 100 healthy test cases featuring 4 independent human segmentations each.
• Introduction of a Fractional Threshold (FT) metric to quantify robustness relative to human performance.
• Assessment of various deep learning models and implementation of an active learning approach for human-in-the-loop revision of uncertain predictions.

Main Results:

• The best 3D U-Net model achieved high overlap metrics (DSC: 0.88, NSD: 0.77), comparable to human readers (DSC: 0.89, NSD: 0.75).
• However, the Fractional Threshold (FT) metric revealed persistent model variability compared to human performance.
• Active learning with human-in-the-loop revision dramatically improved robustness (FT to 0.99) with minimal time investment per case (1.54 minutes), reducing workload by 23-fold.

Conclusions:

• Automated pancreas segmentation offers workload reduction but is limited by unpredictable failures in challenging cases.
• Active learning strategies are crucial for enhancing model reliability and bridging the performance gap between AI and human experts.
• Integrating active learning represents a significant step towards robust and clinically deployable AI for medical image segmentation.