Estimation of inferential uncertainty in assessing expert segmentation performance from STAPLE

Area of Science:

• Medical image analysis
• Computational pathology

Background:

• Evaluating image segmentation quality and expert variability is challenging.
• The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm estimates segmentation performance but lacks uncertainty information.
• Understanding uncertainty is crucial for interpreting performance and determining data sufficiency.

Purpose of the Study:

• To develop a novel algorithm for estimating inferential uncertainty of performance parameters in binary image segmentations.
• To provide confidence intervals for segmentation performance estimates.
• To guide the determination of necessary data size and image segmentation numbers for accurate performance characterization.

Main Methods:

• Developed a new algorithm based on the Expectation Maximization (EM) framework for the STAPLE algorithm.
• Derived the algorithm from general covariance matrix estimation theory for EM algorithms.
• Estimated bounds on performance using the observed Information Matrix.

Main Results:

• Successfully estimated confidence intervals for expert segmentation performance parameters.
• Demonstrated the algorithm's utility on simulated images and neonatal brain MRIs.
• Investigated the impact of the number of experts and image size on performance bounds.

Conclusions:

• The new algorithm effectively estimates inferential uncertainty for segmentation performance parameters.
• It enables the determination of optimal data size and image segmentation numbers for reliable performance assessment.
• This work enhances the interpretation and reliability of image segmentation quality evaluations.