Comparison between exact and parametric distributions of multiple inter-raters agreement coefficient

The Global Kappa statistic can be used for measuring the degree of agreement between more than two raters. In this paper, we give simple computation formulas in the case of dichotomic and no missing responses in order to describe the exact distribution of Kappa for various sets defined by the number of raters, of different examinations and of positive responses. By programming those computations with the SAS software package, we were able to compare the exact and parametric distributions of Kappa. Although the correlation between parameters is high, a linear regression analysis shows that a strong imbalance between positive and negative responses can lead to misinterpretation of the results when using the parametric method.