Do process evaluations open up the 'black box' of implementation interventions in health care? A scoping review

Process evaluations are considered an essential component in conducting and reporting complex interventions, such as those studied in randomised controlled trials (RCTs) of implementation interventions, to explain the effect of implementation interventions. Given the growth of RCTs of implementation interventions with embedded process evaluations, it is timely to review the explanatory learnings to date. This scoping review explores process evaluations of RCTs of implementation interventions to examine how studies are conducted and what insights can be offered about how and why implementation interventions achieve (or not) their intended impacts.

The scoping review was conducted in accordance with the JBI methodology. MEDLINE, CINAHL, Scopus, Web of Science and PsycINFO were searched. Articles were screened and data were extracted by two independent reviewers.

Of the 5857 studies screened, 81 process evaluations were included. Two process evaluations reported on the same trial, resulting in a final number of n = 80 independent studies. Half of studies (48%) reported on implementation trials with no demonstrated effect on the primary outcome (null), while n = 32 (40%) reported on trials where the intervention group demonstrated positive changes in the primary outcome (positive). Seven studies (9%) had mixed findings and n = 3 (4%) studies had no reported trial outcomes. When comparing process evaluation findings from positive and null trials, few discernible patterns that clearly explained the difference in outcomes were identified. Education and training was the most common strategy used in implementation interventions, yet one of the most common implementation barriers reported related to knowledge and self-efficacy, which could indicate a misalignment. Availability of resources was the most prominent barrier for both positive and null trials and there was little evidence that implementation interventions were tailored to context despite prominent barriers and enablers at the inner and outer setting level.

Process evaluation studies embedded in RCTs of implementation interventions are recommended as an important method to explain whether and how interventions produce their intended effect. This review suggests a need to further optimise the design and evaluation of implementation interventions, including the conduct and reporting of process evaluations, to continue advancing the science and practice of implementation.

Protocol published in Open Science Framework, May 10 2022 (Collyer et al., Process evaluations in randomised trials of implementation interventions in health care: a scoping review protocol. In Open science framework, 2022).