The dependence of kappa on attribute prevalence when assessing the repeatability of questionnaire data

The objective of this paper was to illustrate the dependence of the kappa statistic on the underlying population prevalence of a measured outcome, and the related difficulties in interpreting kappa values for assessing the repeatability of survey data. An example was taken from a test-retest study of the repeatability of a questionnaire administered to dairy producers in Ontario, Canada. Ten producers completed two questionnaires, which contained numerous identical questions. Kappa was used to assess repeatability between the questionnaires for questions with responses which were measured on a dichotomous scale. The magnitude of the kappa statistic is shown to be a function of the prevalence of the trait measured by a question as well as the number of discordant responses.