Confidence intervals around Bayes Cost in multi-state diagnostic settings to estimate optimal performance

A critical feature of diagnostic testing is correctly classifying subjects based upon specified thresholds of some measure. The commonly employed Youden index determines a test's optimal thresholds by maximizing the correct classification rates for a diagnostic scenario. An alternative to the Youden index is the cost function, Bayes Cost (BC). BC determines a test's optimal setting by minimizing the sum of all misclassification rates from the test. Unlike the Youden index, BC can consider a priori costs of all the diagnostic outcomes including class specific misclassifications regardless of the number of classes. Delta method approximate confidence intervals around BC are derived under the assumption of normally distributed classes as a means for quantifying a test's performance and comparing classifiers at their optimal settings in a multi-state diagnostic framework. A simulation study is conducted to demonstrate the performance of the derived confidence intervals that are found to perform well, especially for sample sizes of 50 or larger in each diagnostic class. Finally, the proposed methods are applied to a four-class breast tissue classification problem, where four possible discriminatory features are compared under varying decision cost structures. Using the confidence intervals around BC, the best feature for classification is selected, and the optimal thresholds and their 95% confidence intervals are determined.