Confidence intervals for performance assessment of linear observers

Area of Science:

• Medical Physics
• Image Analysis
• Statistical Modeling

Background:

• Linear computerized observers are crucial for optimizing image reconstruction and acquisition in medical imaging, particularly for tasks like lesion detection.
• Assessing image quality requires metrics that account for complex factors such as resolution, noise, and task-specific performance.
• Existing methods for estimating confidence intervals for receiver operating characteristic (ROC) curves often lack exact coverage probabilities.

Purpose of the Study:

• To develop exact confidence interval estimators for figures of merit used to evaluate linear observers.
• To demonstrate the application of these estimators in the context of x-ray computed tomography (CT) for image quality assessment.
• To provide a tool for optimizing parameters in image reconstruction algorithms and data acquisition geometries.

Main Methods:

• Introduced a point estimator for the observer signal-to-noise ratio (SNR) and determined its sampling distribution.
• Constructed exact confidence intervals based on the derived sampling distribution under specific statistical assumptions (normal distribution of ratings, equal variances).
• Developed a computational routine for easily calculating these confidence intervals.

Main Results:

• The new confidence intervals offer exactly known coverage probabilities when data assumptions are met, surpassing existing methods.
• Demonstrated applicability to common ROC summary measures, including area under the curve (AUC), with real x-ray CT data.
• Showed significant robustness to deviations from equal variance assumptions and quantified potential reductions in interval length (up to 35%) with additional data.

Conclusions:

• Exact confidence intervals can be constructed for ROC curves and summary measures for fixed linear observers in binary discrimination tasks.
• These intervals are valuable for optimizing parameters in image reconstruction and acquisition, especially in x-ray CT.
• The method provides a statistically rigorous tool for image quality evaluation under specific, yet relevant, conditions.