[Analyzing interrater agreement for categorical data using Cohen's kappa and alternative coefficients]

Area of Science:

• Rehabilitation research
• Psychometrics

Background:

• Ratings are frequently used in rehabilitation research for patient assessment.
• Interrater agreement is crucial for reliable and meaningful data interpretation.
• Cohen's kappa is a common measure for quantifying interrater agreement.

Purpose of the Study:

• To characterize the properties of Cohen's kappa.
• To clarify conditions for its appropriate application.
• To address potential misconceptions and misleading interpretations of kappa values.

Main Methods:

• Analysis of Cohen's kappa properties and application conditions.
• Description of alternative evaluation strategies for dichotomous rating formats.
• Demonstration of weighted Cohen's kappa (omega) for polytomous rating formats.

Main Results:

• Cohen's kappa values can be affected by factors independent of rating quality.
• Misleading interpretations of kappa can arise from not considering its properties.
• Alternative strategies and weighted kappa offer more valid agreement analyses.

Conclusions:

• Careful consideration of Cohen's kappa properties is essential for accurate interpretation.
• Alternative methods enhance agreement analysis validity in rehabilitation research.
• Weighted kappa provides a tool for analyzing polytomous rating formats.