Using localization data from image interpretations to improve estimates of performance accuracy

A recently developed model uses the localization of abnormalities on images to improve statistical precision in measuring detection accuracy Az, the area below an observer's receiver operating characteristic (ROC) curve for ratings of sampled normal and abnormal cases. This study evaluated that improvement by investigating how much the standard error of estimated Az decreased when the statistical analysis included localization data. Comparisons of analyses with vs without localizations were made for: 1) the estimates of Az from observers' rating ROC curves for nodular lesions on clinical chest films and liver CT scans; 2) the probability of correct choices between paired samples of normal and abnormal cases (equivalent to Az); and 3) the sampling distributions of Az measured in Monte Carlo simulations of 2,000 independent rating experiments. Localization information considerably improved the precision of Az estimates, particularly when detection accuracy was low (Az approximately 0.60). These data provided roughly the same benefits in estimation precision as would two-to-fourfold increases in the sizes of both 1) the samples of positive and negative cases and 2) the observer samples used to estimate Az means.