Assessing the inter-rater agreement for ordinal data through weighted indexes

Area of Science:

• Statistics
• Biomedical Research
• Ordinal Data Analysis

Background:

• Inter-rater agreement is crucial for statistical theory and biomedical applications, especially with ordinal variables.
• Existing methods like Cohen's and Fleiss' kappa can exhibit paradoxical behavior, complicating interpretation.
• There is a need for robust measures of agreement that avoid these paradoxes.

Purpose of the Study:

• To propose a novel modification of Fleiss' kappa for assessing inter-rater agreement with ordinal variables.
• To address the paradoxical behavior observed in traditional kappa statistics.
• To generalize the proposed statistic for broader applicability, including bivariate cases.

Main Methods:

• Development of a modified Fleiss' kappa statistic (s*) for ordinal variables.
• Utilizing Monte Carlo simulations for hypothesis testing and confidence interval calculation (percentile and bootstrap-t).
• Demonstration of the normal asymptotic distribution of the proposed statistic.

Main Results:

• The proposed statistic effectively assesses inter-rater agreement for ordinal data without paradoxical behavior.
• Monte Carlo simulations validated the statistical properties and reliability of the new measure.
• The method was successfully applied to a real-world dataset on cervical cancer classification.

Conclusions:

• The novel statistic provides a reliable and interpretable measure for inter-rater agreement with ordinal variables.
• This advancement improves the analysis of agreement in complex biomedical datasets.
• The generalization to bivariate cases expands its utility in statistical applications.