Conditional inference for predictive agreement

We introduce the concept, and a measure of predictive agreement, tau, for two raters classifying items into q categories. The measure is based on the linear combination of log odds ratios from 2 x 2 subtables of a q x q cross-classification table. We show that analysis procedures for this measure, and transforms of it, can be based on conditional likelihood procedures. These procedures are exact, which is particularly helpful because of the small tabular cell frequencies which can typically arise with agreement data. To illustrate the advantages of the methodology, examples of typical agreement data arising from medical studies are considered. We demonstrate that the conditional likelihood function portrays the available sample information about tau, often more appropriately than the maximum likelihood estimate and an associated standard error. We highlight the value of combining information via likelihoods in an example involving 24 2 x 2 tables. An example involving three categories is used to illustrate that the methodology for the overall agreement measure can be adapted to examine relative agreement between pairs of categories.