Asymptotic variability of (multilevel) multirater kappa coefficients

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Agreement studies are crucial across scientific fields.
• Multirater kappa coefficients quantify inter-rater reliability for categorical data.
• Existing statistical software often limits standard error calculations for kappa coefficients, hindering confidence interval construction.

Purpose of the Study:

• To provide simple analytic formulas for the standard error of multirater kappa coefficients in the general case.
• To extend these formulas to handle multilevel data structures.
• To offer guidelines for selecting appropriate multirater kappa coefficients.

Main Methods:

• Derivation of analytic formulas for standard error using matrix algebra.
• Extension of formulas to accommodate multilevel data.
• Implementation of the methods in the R package "multiagree".

Main Results:

• Novel analytic formulas for the standard error of multirater kappa coefficients are presented.
• The developed formulas are applicable to both standard and multilevel data structures.
• The R package "multiagree" provides a practical tool for implementing these calculations.

Conclusions:

• The study provides essential tools for accurate confidence interval estimation in agreement studies.
• The developed methods enhance the statistical rigor of inter-rater reliability assessments.
• Guidelines are offered to aid researchers in choosing the most suitable kappa coefficient for their data.