Kappa as an index of reproducibility: distribution under the null-hypothesis

The authors report a Monte-Carlo study of the properties of the weighted Kappa (Kw) statistic under the null-hypothesis and the assumption of equal marginal distributions, two hypotheses required in order to use Kw as an index of reproducibility. Different factors that may affect the Kw distribution are investigated: sample size, choice of the set of weights, number of levels of the rating scale and shape of the marginal distribution. Convergence towards the asymptotic distribution is studied and confidence limits are derived. For these latter, except in the uniform marginal distribution case, the use of values obtained by simulation instead of asymptotic estimates appears to be necessary for sample sizes of 30 or less.