Methods for estimating areas under receiver-operating characteristic curves: illustration with somatic-cell scores in...

The aim of this study was to demonstrate receiver-operating characteristic (ROC) methodology in the context of bovine intramammary infection (IMI). Quarter somatic cell scores (SCS) were available to evaluate quarter IMI, and the final IMI diagnosis was made from milk bacteriologic cultures. Data consisted of 11,453 quarter-milk samples collected on 2084 clinically healthy cows located in 154 Belgian herds. Bacteriological analyses showed 16.2%, 7.2%, and 11.9% of quarters infected with coagulase-positive Staphylococcus spp., Streptococcus agalactiae, and coagulase-negative Staphylococcus spp., respectively. The ROC curve indicated all the combinations of sensitivity and specificity that quarter SCS was able to provide as a test to identify quarter IMI. Among parametric, semi-parametric, and non-parametric methods to estimate area under ROC curves, the parametric method seemed the least appropriate for analyzing SCS in this study. With the non-parametric method, the total area under the ROC curves showed quarter SCS could identify quarter IMI with an overall accuracy of 69%, 76%, and 59% for coagulase-positive Staphylococcus spp., S. agalactiae, and coagulase-negative Staphylococcus spp., respectively. Parametric and non-parametric statistical tests showed that overall SCS diagnostic capability was significantly (p<0.01) different from chance and was different (p<0.01) across the three bacteria. However, the SCS thresholds yielding the highest percentage of quarters correctly classified as infected (for the observed prevalence and for equal costs assigned to false-positive and false-negative results) were so high that they had no practical value. The major advantage of ROC analysis is the comprehensive description of the discrimination capacity of SCS for all possible choices of critical values. The major disadvantage is the dependency upon the gold standard used for the final diagnosis--but recent improvements of the methodology will correct the problem.