Related Experiment Videos
Reader variance in ROC studies--generalizability to reader population at high and low performance levels
David Gur1, Andriy I Bandos, Amy H Klym
1Department of Radiology, School of Medicine, University of Pittsburgh, Pittsburgh, PA 15261, USA. gurd@upmc.edu
Academic Radiology
|July 18, 2006
Summary
Reader variability in diagnostic performance is lowest at very high and very low average performance levels. This finding is crucial for understanding the generalizability of receiver operating characteristic (ROC) study results.
Area of Science:
- Radiology
- Medical Imaging Analysis
- Statistical Modeling in Medicine
Background:
- Receiver operating characteristic (ROC) studies are vital for evaluating diagnostic test performance.
- Understanding reader variability is essential for interpreting and generalizing ROC study findings.
Purpose of the Study:
- To investigate how reader variability in diagnostic performance changes with average performance levels in ROC studies.
- To identify the relationship between average performance and reader variance components.
Main Methods:
- Reanalyzed ROC rating data from 12 observers detecting interstitial disease and pneumothorax.
- Selected case subsets based on abnormality subtlety and difficulty of assessment.
- Estimated reader variance component using a bootstrap approach.
Main Results:
- Reader variance decreased with increasing Area Under the Curve (AUC) for pneumothorax detection (84% to 97%).
- Reader variance increased with increasing AUC for interstitial disease detection (59% to 87%).
- A non-monotonic relationship was observed between reader variance and AUC.
Conclusions:
- Reader variance component is not monotonically dependent on average diagnostic performance.
- The lowest reader variability occurs at both very high and very low average performance levels.
- This non-monotonic relationship impacts the generalizability of ROC study results.