Higher-order kappa-type statistics for a dichotomous attribute in multiple ratings

Area of Science:

• Statistics
• Psychometrics
• Data Analysis

Background:

• Assessing inter-rater reliability is crucial for data quality.
• The kappa coefficient is widely used but limited to pairwise agreement.
• Evaluating consensus among multiple raters requires advanced statistical methods.

Purpose of the Study:

• Introduce canonical moments as a novel metric for rating consensus.
• Extend the concept of agreement beyond pairwise comparisons.
• Provide a framework for analyzing multiple binary ratings.

Main Methods:

• Define a sequence of canonical moments derived from rating patterns.
• Establish the relationship between canonical moments and the kappa coefficient.
• Demonstrate the application of higher canonical moments for multi-rater consensus.

Main Results:

• The traditional kappa coefficient represents the second canonical moment.
• Higher canonical moments capture more complex patterns of agreement.
• Canonical moments offer a robust method for evaluating consensus in multiple binary ratings.

Conclusions:

• Canonical moments provide a powerful extension to traditional agreement metrics.
• This sequence offers a nuanced understanding of inter-rater reliability.
• The framework is applicable to various fields requiring consensus evaluation.