Creating a Statistical Analysis Plan to Continually Evaluate Intervention Adaptations that Arise in Real-World Implementation

Evidence-based health interventions are frequently translated into real-world settings where practical needs drive changes to intervention protocols. Due to logistical and resource constraints, these naturally arising adaptations are rarely assessed for comparative effectiveness using a randomized trial. Nevertheless, when observational data are available, it is still possible to identify beneficial adaptations using statistical methods that adjust for differences among intervention groups. As implementation continues and more data are collected and assessed, we also require analysis methods that ensure low statistical error rates as multiple comparisons are made over time. This paper describes how to create a statistical analysis plan for evaluating adaptations to an intervention during ongoing implementation. This can be done by combining methods commonly used in platform clinical trials with methods used for real-world data. We also demonstrate how to use simulations based on previous data to decide the frequency with which to conduct statistical analyses. The illustration uses data from large-scale implementation of a school-based resilience and skill-building preventive intervention to which several adaptations were made. The proposed statistical analysis plan for evaluating the school-based intervention has potential to improve population-level outcomes as implementation scales up further and additional adaptations are anticipated.