Does Reproducibility Drive Clinical Accuracy?

To develop a stochastic model relating measurement uncertainty, including reproducibility, to clinical accuracy, as demonstrated by the receiver operating characteristic curve.

A model is developed based on the symmetric case of the well-known binormal distribution. The overall distribution is partitioned further into analytical and biological components based on assumptions derived from the Cotlove criterion. Explicit mathematical solutions are derived and further verified by Monte Carlo analyses.

The model demonstrates that tests with analytical error that conforms to the classic Cotlove criterion can achieve receiver operating characteristic curves with areas under the curve of 0.68 to 0.76 and Youden indices of 0.26 to 0.38 but have overall agreement for duplicate measurements of only 80% to 82%. Furthermore, the analytically accurate agreement is only 75% to 78%, and the clinically accurate agreement is only 50% to 60%.

The model suggests that assays may have reasonable clinical accuracy despite having reproducibility of less than 85%. Imperfect assays can substantially improve medical decision-making. The findings must be interpreted with caution given the binormal assumptions, but such assumptions are often useful as a first approximation. Practicing pathologists should feel comfortable performing semiquantitative assays shown to have a strong biological association with clinical outcome.