Ensemble-based uncertainty enables large-scale 3D segmentation quality control in population imaging

Modern Deep-Learning facilitates automatic segmentation in large imaging studies. To prevent segmentation errors from impacting data analysis, automated quality control (QC) for filtering segmentations and identifying errors (segmentation selection) is essential. QC must be fast and require minimal expert delineations for application to various ROIs in large datasets. Uncertainty quantification methods capable of estimating segmentation error without ground truth offer a promising foundation. However, their implicit error estimation may lack accuracy, making suitability for segmentation selection unclear. Unfortunately, commonly employed evaluation measures cannot capture their segmentation selection capability. Furthermore, selecting or rejecting segmentations based on uncertainty alone can be non-trivial. We propose a new method for efficient segmentation QC in large imaging cohorts. We comprehensively evaluate uncertainty-based error estimators with respect to key goals of large-scale QC, specifically assessing segmentation selection capability through the area under the risk-coverage-curve. To optimize selection and rejection, we employ a binary search for decision thresholds, balancing the risk of including errors and unnecessarily rejecting segmentations. In >11 000 MRI liver and lung segmentations in the German National Cohort, error estimation using the pairwise Dice of ensemble outputs achieved excellent QC, including segmentation selection. Numerous manually hard-to-detect errors and abnormalities (n>400) were identified during high-quality result selection, measurably restoring integrity of volume and fat measurements. QC was fast (0.29-4.48s/case) and required minimal annotated image data (n<125). Our method can help tackle the challenge of reliable automated QC in large-scale segmentation studies.