The standard error of Cohen's Kappa

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Cohen's Kappa is a widely used statistic for measuring inter-rater reliability.
• Existing methods for calculating the standard error of Cohen's Kappa have limitations, particularly under certain conditions.

Purpose of the Study:

• To develop a more accurate standard error for Cohen's Kappa, especially when conditional on observed margins.
• To provide explicit formulas and procedures for both 2x2 and general r x r tables.
• To propose a log-linear model for approximate confidence intervals of Kappa.

Main Methods:

• Derivation of an explicit standard error formula for 2x2 tables.
• Development of a procedure for general r x r tables using a parsimonious log-linear model.
• Comparison of proposed methods with standard and exact conditional results through numerical illustrations.

Main Results:

• The standard error under the null hypothesis should only be used when the null is plausible.
• The usual standard error formula is generally appropriate, except for large negative Kappa values.
• Kappa estimates exhibit non-symmetric distributions in small samples, favoring transformed confidence intervals.

Conclusions:

• The proposed standard error and confidence interval methods offer improvements for Cohen's Kappa analysis.
• Researchers should be cautious when applying standard error formulas under the null hypothesis.
• For small sample sizes, using transformations of Kappa for confidence intervals is recommended for better accuracy.