Interrater agreement of visual analysis in single-subject decisions: quantitative review and analysis

The empirical literature examining the interrater agreement for visual analysis of single-subject data was examined using the technique of meta-analysis. Aggregated study results revealed an overall interrater agreement value of .58 for 14 studies representing a total of 789 raters. Additional analyses revealed that the interrater agreement values did not vary across raters based on previous experience or level of expertise. Characteristics of the graphs, including whether the data were hypothetical or real, or whether designs were the simple AB format or more complex (ABAB) designs, did not significantly influence the interrater agreement values. The only graphic characteristic related to increased interrater agreement was the presence of a trend (regression) line. Trend lines appeared to improve interrater agreement if the raters were familiar with their method of computation and interpretation. Simply including a trend line in the graphs produced only a marginal improvement in interrater agreement.