The dependence of Cohen's kappa on the prevalence does not matter

The dependence of Cohen's kappa on the prevalence has been a major concern in the literature. Indeed, it indicates a serious limitation with respect to comparing kappa-values among studies with varying prevalences.

The basic arguments used by different authors are reviewed.

Two types of dependence can be distinguished: a dependence on the observed marginal prevalences and a dependence on the prevalence of a latent binary variable, representing the true status. The first dependence is simply a consequence of the purpose of kappa, which is to improve the interpretation of agreement rates, and so does not constitute a real argument against kappa. The second occurs only if one can change the prevalence without changing sensitivity and specificity. Typically, in agreement studies a change in prevalence implies also a change in sensitivity and specificity, and we show that in such a framework the dependence on the prevalence becomes negligible.

We should stop criticizing kappa for its dependence on the prevalence. Instead, we should focus on its dependence on the composition of the population with respect to subjects easy or difficult to agree on.