Validation and statistical power comparison of methods for analyzing free-response observer performance studies

Area of Science:

• Medical Imaging Analysis
• Statistical Modeling
• Diagnostic Accuracy Studies

Background:

• Free-response data analysis is crucial in medical imaging for evaluating diagnostic performance.
• Existing statistical methods require validation for their power and applicability.
• Computer-aided detection (CAD) systems necessitate robust evaluation techniques.

Purpose of the Study:

• To validate and compare the statistical powers of various methods for analyzing free-response data.
• To assess the performance of different statistical approaches using a search-model-based simulator.
• To provide recommendations for optimal analysis methods based on observer type (human vs. CAD).

Main Methods:

• Developed a free-response data simulator modeling single-reader, dual-modality, or dual-observer scenarios.
• Employed a variance components model to account for intracase and intermodality correlations.
• Investigated the null hypothesis validity and statistical powers of ROC, JAFROC, JAFROC-1, IDCA, and NP methods for simulated human observers and CAD algorithms.

Main Results:

• All tested methods demonstrated valid null hypothesis behavior across various simulator parameters.
• For human observers, the statistical power ranking was JAFROC-1 > JAFROC > (IDCA ≈ NP) > ROC.
• For CAD algorithms, the ranking was (NP ≈ IDCA) > (JAFROC-1 ≈ JAFROC) > ROC, with top methods showing double the power of the lowest.

Conclusions:

• JAFROC-1 is recommended for analyzing free-response data from human observers, including those using CAD assistance.
• The nonparametric (NP) method is recommended for evaluating computer-aided detection (CAD) algorithms.
• The choice of method significantly impacts the statistical power in free-response diagnostic accuracy studies.