[Analyzing interrater agreement for categorical data using Cohen's kappa and alternative coefficients]

Within rehabilitation research ratings are one of the most frequently used assessment procedures. For example, therapists frequently make categorical judgements aiming to get information whether specific patient characteristics prevail or not (dichotomous rating format) or which of several alternatives holds for a patient (polytomous rating format). Interrater agreement is an important prerequisite to ensure that reliable and meaningful information concerning patients' state can be inferred from the data obtained. Cohen's kappa (Cohen's kappa) is the most frequently used measure to quantify interrater agreement. The properties of Cohen's kappa are characterized and conditions for the appropriate application of kappa are clarified. Because sometimes specific properties of kappa are not appropriately considered, misleading interpretations of this measure may easily arise. This is the case because the value of Cohen's kappa is affected by information aspects that are independent of the quality of the rating process. In order to avoid such misconceptions, alternative evaluation strategies are described for dichotomous rating formats which enhance agreement analysis and thus ensure a more valid interpretation. In addition, it is shown how weighted Cohen's kappa (omega) may be used to analyze polytomous rating formats.