Comparison of receiver operating characteristic and forced choice observer performance measurement methods

Area of Science:

• Medical Imaging
• Psychophysics
• Observer Performance

Background:

• The receiver operating characteristic (ROC) method is widely used in medical imaging for evaluating human observer performance.
• It is often viewed as an end in itself, rather than a tool for assessing decision-making accuracy.
• Accurate and precise performance estimates are desired with limited data.

Purpose of the Study:

• Introduce medical physicists to the forced choice psychophysical measurement technique.
• Compare the forced choice and ROC methods, focusing on sampling statistics.
• Provide guidance on selecting the optimal method for observer performance assessment.

Main Methods:

• Comparative analysis of the rating scale ROC method and the forced choice method.
• Consideration of sampling statistics and experimental constraints.
• Evaluation of performance under different conditions, including synthetic images.

Main Results:

• The rating scale ROC method is preferable when image count is limited and the forced choice method has few alternatives.
• The forced choice method can be superior under specific conditions, such as with synthetic images.
• Sampling statistics significantly influence the choice and effectiveness of each method.

Conclusions:

• The choice between ROC and forced choice methods depends on experimental constraints and objectives.
• Both methods are valuable for assessing observer performance in medical imaging.
• Understanding the statistical properties of each method is crucial for accurate results.