Using Mixed Methods to Evaluate Risk Minimisation Programs in Europe and the USA: An Innovative Blueprint

Significant methodological shortcomings have been documented to date in risk minimisation program evaluations for medicinal products, including overreliance on survey methods alone. Recently updated guidances from the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) recommend the use of frameworks and mixed methods designs to improve the rigor of these assessments.

The purpose of this paper was to exemplify how a mixed methods approach, guided by an implementation science framework, can be used to design the evaluation of a risk minimisation program.

We selected the Practical, Robust, Implementation and Sustainability Model (PRISM) as the implementation science framework to guide our mixed methods approach. PRISM provides a comprehensive and systematic approach to measuring the key domains relevant to the implementation and outcomes of a risk minimisation program. We mapped the PRISM domains to the evaluation dimensions described in the EMA and FDA guidances. We then specified a mixed methods evaluation design and data collection methods using a fictitious risk minimisation program as a case study for illustrative purposes.

On the basis of our case study, we developed quantitative and qualitative measures, including specific items for surveys and interviews, for both formative and summative evaluations. For both the formative and summative evaluations, measures focussed on assessing (1) contextual factors that could affect program implementation and impact and (2) outcomes including implementability and acceptability as well as degree of program reach, adoption, implementation, effectiveness and maintenance.

Mixed methods, guided by a well-established implementation science framework, can be applied to ensure comprehensive formative and summative evaluations that provide fit-for-purpose information that may inform regulatory decision-making.