A contaminated binormal model for ROC data: Part I. Some interesting examples of binormal degeneracy

Receiver operating characteristic (ROC) data with false-positive fractions of 0 are often difficult to fit with standard ROC methods and are sometimes discarded. Some extreme examples of such data were analyzed to evaluate the nature of these difficulties.

Rating reports of fracture for single-view ankle radiographs were analyzed with the binormal ROC model and with two ROC models that keep the ROC curve from crossing the chance line. Because fractures were almost never reported that were not present, some views and locations yielded only ROC points with false-positive fractions of 0, while others yielded at least one ROC point with a non-0 false-positive fraction.

The models tended to yield ROC areas close to or equal to 1. ROC areas of 1 imply a true-positive fraction close to 1; yet the data contained no such fractions. When all false-positive fractions were 0, the true-positive fraction could be much higher for one view than another for all observers. ROC areas gave little or no hint of these unmistakable differences in performance.

These data challenge the validity and robustness of current ROC models. A key aspect of ankle fractures is that some may be visible on one view but not at all visible on another.