To judge the judges--kappa, ROC or what?

When two raters or two classification methods are compared, it is of interest both to describe to which extent they give identical results and to find out whether they differ in a systematic way. More often than not, the kappa coefficient (K) is used-inspite of well known short comings concerning this characteristic and the difficulties to display the results in a digestible way. For the comparison of two dichotomous classification methods it will be demonstrated that the confidence interval for the systematic difference between the methods is related to the kappa coefficient so that the standard error for the difference decreases with increasing kappa values. A graphical presentation utilizing this fact is proposed. The confidence interval is compared with the length had the two methods been completely unrelated and with the length if K had the largest possible value, given the actual value of the difference. The procedure is extended to the comparison of two methods with ordinal scales and a number of comments regarding the comparison between different classification methods will be given. Finally, a real life example comparing two pathological classification methods is used as a basis for demonstrating a number of different ways to display the results from this type of studies by means of elementary statistical methods.