Central tendency and matched difference approaches for assessing interrater agreement

Area of Science:

• Psychology
• Statistics
• Social Sciences

Background:

• Interrater agreement is crucial for group-level research.
• Existing methods may not adequately assess agreement across multiple groups or distinguish real from pseudo agreement.

Purpose of the Study:

• To develop and validate novel procedures for assessing interrater agreement within groups.
• To introduce a "central tendency approach" and a "matched difference approach" for robust agreement analysis.

Main Methods:

• Study 1: Developed a "central tendency approach" using bootstrapped confidence intervals for mean group agreement (rWG, AD, ICC(1)).
• Study 2: Developed a "matched difference approach" using random group resampling to assess real versus pseudo agreement.
• Utilized simulated and real data, along with a new multilevel data generation model, for validation.

Main Results:

• Demonstrated the accuracy and practical utility of the proposed procedures in both simulated and real-world data.
• The new inference procedures provide enhanced information on population-level interrater agreement compared to current practices.

Conclusions:

• The developed methods offer improved assessment of interrater agreement in group-level studies.
• These procedures can lead to better data aggregation decisions and more accurate interpretation of findings.