Investigating the effects of ties on measures of concordance

The concordance between predicted and observed outcomes, referred to as the C index or C statistic, is frequently used to quantify the discriminatory ability of a prognostic model. It also commonly serves as a basis for distinguishing predictive strength between different models. Two alternative formulations of the C index are widely used, one completely excluding ties from the computation (Ctied,out) and the other including ties (Ctied,in). However, there has been little research concerning the effects of ties on these two measures. In this paper we characterize changes in the performance of Ctied,in and Ctied,out for progressively less coarse (or more coarse) partitions of the data. Our theoretical and simulation results show that both measures can be heavily dependent on the number of tied pairs and their results can be substantially divergent. We examine potential ambiguities that can occur when the two measures diverge. In the presence of a substantial proportion of tied pairs, we recommend that both C indices be computed as lower and upper bounds, and that, when feasible, the C indices should be computed with respect to partitions corresponding to the same percentiles for the models being compared. When it is desired to estimate the overall concordance, the average value of the two C indices might also be considered. We use the data from the Hemodialysis Clinical Trial to illustrate our evaluation and methods.