Estimation of inferential uncertainty in assessing expert segmentation performance from STAPLE

Area of Science:

• Medical image analysis
• Computational imaging
• Biomedical engineering

Background:

• Evaluating image segmentation quality and expert variability is challenging.
• Simultaneous Truth and Performance Level Estimation (STAPLE) estimates reference segmentations but lacks uncertainty quantification.
• Understanding uncertainty is vital for interpreting performance and determining data sufficiency.

Purpose of the Study:

• To develop a novel algorithm for estimating inferential uncertainty in segmentation performance parameters.
• To provide confidence intervals for performance metrics derived from multiple segmentations.
• To guide the determination of optimal data size and number of experts for accurate segmentation assessment.

Main Methods:

• Developed a new algorithm based on covariance matrix estimation for Expectation-Maximization (EM) algorithms.
• Applied the algorithm to estimate bounds on performance parameters using the observed information matrix.
• Validated the method on simulated images and neonatal brain MRI datasets.

Main Results:

• Successfully estimated confidence intervals for expert segmentation performance parameters.
• Demonstrated the algorithm's ability to quantify uncertainty in segmentation quality assessment.
• Investigated the influence of the number of experts and data size on performance parameter bounds.

Conclusions:

• The proposed algorithm effectively estimates inferential uncertainty in segmentation performance.
• Confidence intervals aid in the interpretation of segmentation generator performance.
• The method helps determine the necessary data size and number of segmentations for reliable performance assessment.