Comparison between exact and parametric distributions of multiple inter-raters agreement coefficient

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• The Global Kappa statistic is a valuable tool for assessing inter-rater reliability when more than two raters are involved.
• Accurate interpretation of Kappa statistics is crucial in various fields, including medical research, psychology, and social sciences.

Purpose of the Study:

• To provide simple computation formulas for the exact distribution of the Global Kappa statistic.
• To compare the exact distribution with the parametric distribution of Kappa under various conditions.
• To identify potential limitations of parametric methods in the presence of response imbalance.

Main Methods:

• Derivation of exact computation formulas for the Global Kappa statistic with dichotomous and complete data.
• Implementation of these formulas using SAS software for computational analysis.
• Comparison of exact Kappa distributions with parametric approximations using linear regression analysis.

Main Results:

• The study successfully derived and programmed formulas for the exact distribution of the Global Kappa statistic.
• A high correlation was observed between parameters of the exact and parametric distributions.
• Linear regression analysis indicated that significant imbalances in positive and negative responses can lead to misinterpretations when relying solely on parametric methods.

Conclusions:

• The exact distribution of the Global Kappa statistic provides a more accurate representation, especially when response distributions are uneven.
• Parametric methods for Kappa statistic may require cautious application in scenarios with substantial response imbalance.
• The findings highlight the importance of considering the exact distribution for reliable inter-rater agreement assessment.