Lessons learned in optimizing recruitment and data collection for a pharmacy-based implementation trial

In Fall 2023, we initiated a National Institutes of Health-funded study to test the effectiveness of implementation strategies to support pharmacy implementation of a COVID-19 vaccine hesitancy counseling intervention. Our stepped-wedge trial compared a standard implementation condition (training and technical assistance) to a virtual facilitation (or one-on-one coaching) intervention. Within 6-8 weeks, problems emerged related to pharmacist engagement and data collection that threatened the study's success.

To describe formative evaluation results for the initial trial year and the strategies we developed and deployed in Fall 2024 to address identified engagement and data collection problems.

The formative evaluation consisted of 5 semistructured key informant interviews with highly-engaged and unengaged pharmacists and a review of coaches' notes. All qualitative data were coded using a rapid template analysis method. To interpret the results and identify solutions, we consulted rural patient and implementation science expert advisory panels. Consensus was reached on study modification recommendations, which received funder approval.

Numerous barriers impacted pharmacist engagement and data collection, such as poor demand for the COVID-19 vaccine, most patients declining to be audio-recorded to assess our primary outcome of pharmacist counseling fidelity, difficulties submitting encounter forms (our secondary outcome), staffing shortfalls, and pharmacist reluctance to approach "very hesitant" patients.

Study modifications included changes to data collection (e.g., shift to using standardized patients for fidelity assessments), pharmacy recruitment, and the standard implementation approach (e.g., more training and tools). These modifications were deployed in the new first year of our implementation trial (August 2024 - April 2025) and we saw major improvements in study engagement and data collection.

Implementation trials often encounter potentially fatal problems due to shifting contexts. Studies can be salvaged if the team takes quick action, in partnership with funders and other constituents, to systematically understand the problems and develop solutions.