Four multi-item interrater agreement options: comparisons and outcomes

Area of Science:

• Psychometrics
• Statistical analysis
• Research methodology

Background:

• Interrater agreement is crucial for ensuring the reliability of data collected by multiple observers.
• Various statistical indices exist to quantify agreement among raters, but their interpretation and application can be complex.

Purpose of the Study:

• To compare four distinct multi-item indices of interrater agreement: intraclass correlation coefficient, within-group interrater agreement, modified within-group interrater agreement, and average deviation index.
• To evaluate the consistency of standards for acceptable agreement across these indices.
• To assess the impact of removing data based on unacceptable interrater agreement values on overall research outcomes.

Main Methods:

• Comparative analysis of four multi-item indices for measuring interrater agreement.
• Evaluation of data from multiple groups using each index.
• Assessment of the effect of data exclusion criteria on study results.

Main Results:

• The four indices provided distinct information regarding rater agreement.
• Inconsistent standards for acceptable agreement were observed across the different indices.
• The average deviation index indicated acceptable agreement in most groups.
• Excluding data due to unacceptable interrater agreement had minimal impact on overall outcomes.

Conclusions:

• Interrater agreement indices are not interchangeable and yield different information.
• Current standards for acceptable interrater agreement are inconsistent.
• The significance of interrater agreement in research may be overestimated, particularly when using certain indices.