The impact of probe variability on brief experimental analysis of reading skills

The purpose of this study was to examine the impact of probe variability on the ability to replicate results in brief experimental analysis (BEA) of reading. In the first phase of the study, 41 first- and second- grade students completed 16 oral reading fluency probes. Calculations of probe difficulty were used to identify Low and High Variability probe sets. In the second phase of the study, the performance of 40 second- through fifth-grade students during two reading interventions was compared. The best-performing intervention for each student in the initial trial was replicated during a second trial for only 43% of students regardless of probe variability. The best-performing intervention was replicated for 60% of students when average performance across two trials was compared. Rules for determining the best-performing intervention in academic BEA should consider the standard error of measurement (SEM) for the probe set to be used, the reliability for absolute decisions using the probe set, and the number of replications relative to SEM needed to adequately demonstrate experimental control.