Estimating maximal measurable performance for automated decision systems from the characteristics of the reference...

We investigate the maximal performance that can be measured for automated binary decision systems in terms of area under the ROC curve (AUC), against a reference standard provided by human readers. The goal is to determine the required characteristics of the reference standard to assess and compare automated decision systems with a given degree of confidence, or, to determine what degree of confidence can be obtained given the characteristics of the reference standard. We modeled the expected value of the AUC that can be measured for a perfect decision system, given a reference standard provided either by a single human reader or by multiple human readers (consensus, majority vote). The proposed model was applied to diabetic retinopathy screening in a dataset of 874 eye fundus examinations graded by three readers. The expected value of the AUC for a perfect decision system was estimated at 0.956 against a single human reader, and 0.990 against a `majority wins' vote of three human readers. The Iowa detection program has reached the maximal performance measurable by a single human reader (0.929, CI: [0.897-0.962]) and is close to the maximal performance measurable by a `majority wins' vote (0.955, CI: [0.939-0.972]).