A comparison of resampling schemes for estimating model observer performance with small ensembles

Area of Science:

• Medical Imaging
• Computer Vision
• Statistical Modeling

Background:

• Objective assessment of image quality relies on statistical analysis of image ensembles.
• Limited ensemble sizes introduce statistical variability, impacting numerical observer performance.
• Resampling strategies are crucial for mitigating variability in observer performance metrics.

Purpose of the Study:

• To compare the performance of different resampling schemes (leave-one-out and half-train/half-test) combined with various model observers (channelized Hotelling observer, channelized linear discriminant, channelized quadratic discriminant).
• To evaluate how ensemble size and resampling strategy affect the accuracy of observer performance rankings, quantified by the area under the ROC curve (AUC).

Main Methods:

• The study employed binary classification tasks using different model observers and resampling schemes.
• Performance was evaluated using the area under the ROC curve (AUC) as the primary metric.
• A large ensemble size (2000 samples per class) served as a gold standard for comparison.

Main Results:

• Observer performance varied significantly based on the chosen resampling scheme and observer type.
• For small ensemble sizes, the channelized Hotelling observer (CHO) with half-train/half-test (HT/HT) showed more accurate rankings than with leave-one-out (LOO).
• The channelized linear discriminant (CLD) observer outperformed the CHO, especially with the LOO scheme at smaller ensemble sizes, demonstrating more robust performance.

Conclusions:

• The channelized linear discriminant (CLD) observer paired with the leave-one-out (LOO) resampling scheme is advantageous when dealing with small image ensemble sizes.
• This combination offers more reliable and accurate performance rankings compared to other tested observer-resampling strategies in data-limited scenarios.
• The findings suggest a preference for CLD with LOO for improved statistical reliability in objective image quality assessment with limited data.