A matrix of kappa-type coefficients to assess the reliability of nominal scales

To investigate the reliability of nominal scales, Kraemer proposed a measurement model from which kappa coefficients could be derived. More recently she suggested a matrix of coefficients as a comprehensive summary of reliability, contrasting this with use of a single summary kappa statistic. The main diagonal of the matrix consists of binary kappa coefficients for each category which measure the reliability of each category relative to all others, while the off-diagonal elements are correlation coefficients for pairs of categories. The off-diagonal elements were suggested as measures of confusion between categories. Schouten also suggested coefficients to assess confusion between pairs of categories, which might be used as alternative off-diagonal elements in a summary matrix. The two types of off-diagonal element will be compared. It will be shown that Schouten's coefficients can be expressed in terms of the parameters of Kraemer's measurement model and that they are more easily interpreted as measures of confusion. First, the maximum value for Schouten's coefficient is one. Secondly, for any pair of categories, Schouten's coefficient equals the proportionate reduction in the probability of classifying a subject in one category of the pair having previously classified them in the other. Thirdly, where the coefficient for a pair of categories is less than the summary kappa statistic, it will be shown that combining these two categories will increase the value of the summary kappa statistic. The methods of analysis are applied to data from a study of the reliability of psychiatric diagnosis and used to identify pairs of classifications between which there is substantial confusion.