A power analysis for fidelity measurement sample size determination

The importance of assessing fidelity has been emphasized recently with increasingly sophisticated definitions, assessment procedures, and integration of fidelity data into analyses of outcomes. Fidelity is often measured through observation and coding of instructional sessions either live or by video. However, little guidance has been provided about how to determine the number of observations needed to precisely measure fidelity. We propose a practical method for determining a reasonable sample size for fidelity data collection when fidelity assessment requires observation. The proposed methodology is based on consideration of the power of tests of the treatment effect of outcome itself, as well as of the relationship between fidelity and outcome. It makes use of the methodology of probability sampling from a finite population, because the fidelity parameters of interest are estimated over a specific, limited time frame using a sample. For example, consider a fidelity measure defined as the number of minutes of exposure to a treatment curriculum during the 36 weeks of the study. In this case, the finite population is the 36 sessions, the parameter (number of minutes over the entire 36 sessions) is a total, and the sample is the observed sessions. Software for the sample size calculation is provided.