Four multi-item interrater agreement options: comparisons and outcomes

This study compared four multi-item indices of interrater agreement: (a) intraclass correlation coefficient, (b) within-group interrater agreement, (c) modified within-group interrater agreement, and (d) average deviation index. Findings included (a) that the different indices of interrater agreement provided different information about the agreement across raters, (b) standards for acceptable agreement are inconsistent for the several indices, (c) the average deviation index showed most groups had acceptable agreement, and (d) removing data from analyses based on unacceptable interrater agreement values does little in the way of affecting overall outcomes. Implications include that interrater agreement indices are not interchangeable and that for research purposes the concern about interrater agreement may be overstated.