Bayesian approaches to the weighted kappa-like inter-rater agreement measures

Area of Science:

• Statistics
• Psychometrics
• Data Analysis

Background:

• Inter-rater agreement measures assess consistency between assessors.
• Ordinal agreement tables use weight functions, sensitive to anomalies like unbalanced structures or grey zones.
• Traditional methods can be impacted by rater assessment behavior.

Purpose of the Study:

• To propose Bayesian approaches for estimating inter-rater agreement measures.
• To improve accuracy and mitigate anomalies in agreement estimation.
• To incorporate prior information on rater behavior and impose order restrictions.

Main Methods:

• Developed Bayesian methods for inter-rater agreement estimation.
• Elicited prior distributions theoretically and practically.
• Conducted Monte Carlo simulations to compare Bayesian and classical approaches.

Main Results:

• Bayesian approaches improve accuracy by incorporating prior information.
• The proposed methods mitigate the impact of anomalies in agreement tables.
• Simulation study assessed classification accuracy for various measures and weights.

Conclusions:

• Bayesian methods offer a robust alternative for inter-rater agreement analysis.
• Recommendations are provided for selecting optimal agreement measures and weights based on table structure and sample size.