Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction | JoVE Visualize

Area of Science:

Behavioral Science
Autism Intervention Research
Measurement in Psychology

Background:

Procedural fidelity, or the extent an intervention is implemented as designed, is crucial for research and practice.
Existing research has not extensively compared how different procedural fidelity measurement methods impact findings.
Accurate measurement is vital to ensure intervention effectiveness and avoid misinterpretations in clinical practice.

Purpose of the Study:

To compare the procedural fidelity of behavior technicians implementing discrete-trial instruction protocols for children with autism using various measurement methods.
To assess how different measurement approaches (global, all-or-nothing, Likert scales, occurrence-nonoccurrence) affect the estimation of fidelity at component and trial levels.
To identify potential overestimation or underestimation of fidelity and discuss implications for research and clinical practice.

Main Methods:

Behavior technicians' adherence to discrete-trial instruction protocols was observed.
Procedural fidelity was measured using: occurrence-nonoccurrence data sheets (individual-component and individual-trial), global fidelity, all-or-nothing, 3-point Likert scale, and 5-point Likert scale methods.
Data were collected on the time required for each measurement method.

Main Results:

At the component level, global, 3-point, and 5-point Likert scales tended to overestimate fidelity and mask errors, unlike the all-or-nothing method.
At the trial level, global and 5-point Likert scales approximated individual-trial fidelity, while the 3-point Likert overestimated and all-or-nothing underestimated fidelity.
The occurrence-nonoccurrence method was most time-consuming; all-or-nothing by trial was least time-consuming.

Conclusions:

The choice of procedural fidelity measurement method significantly impacts the assessment of intervention adherence, potentially leading to false positives or negatives.
Global and Likert scale methods may obscure critical implementation errors, whereas the all-or-nothing method can be overly stringent.
Researchers and practitioners should carefully select measurement tools based on the need for sensitivity, specificity, and efficiency to ensure accurate fidelity assessment.