Evaluating for learning and sustainability (ELS) framework: a realist synthesis

Learning Health Systems (LHS), in which continuous and equitable improvements support optimization of healthcare practices, outcomes, experience, and costs, offer enormous potential for health system transformation. Within the LHS model, evaluation of health innovations assists in question identification, data collection, and targeted action, which facilitates continuous improvement. Evaluation that catalyzes learning may contribute to health innovation implementation, refinement, and sustainability, however, there is little consensus as to why certain evaluations support learning, while others impede it.

Embedded in the implementation science literature, we conducted a realist synthesis to understand evaluative contextual factors and underlying mechanisms that best support health system learning and sustainable implementation of innovations. We sought to understand whether evaluations can 'work' to support learning and sustainability, in which contexts, for whom, and why. Working with an Expert Committee comprised of leaders in evaluation, innovation, sustainability, and realist methodology, we followed a five-stage process of: 1. Scoping the Review, 2. Building Theories, 3. Identifying the Evidence, 4. Evidence Selection and Appraisal, and 5.

Our Review Team and Expert Committee participated in iterative cycles of results interpretation and feedback.

Our synthesis includes 60 articles capturing the mechanisms and contextual factors driving learning and sustainability through evaluation. We found that evaluations that support learning and sustainability incorporate favourable organizational preconditions and focus on implementing rapid cyclical feedback loops that contribute to a culture of innovation and evaluation sustainability. Our findings have been organized into 6 Context-Mechanism-Outcome Configurations (CMOCs): 1. Embracing Risk & Failure; 2. Increasing Capacity for Evaluation; 3. Co-Producing Evaluation; 4. Implementing Learning Feedback Loops; 5. Creating Sustainability Culture; and 6. Becoming a Learning Organization. We have also translated findings into a series of Action Strategies for evaluation implementation to support health systems learning and sustainability.

We identified key contextual factors and underlying mechanisms that make evaluations 'work' (or 'not work') to support learning and sustainability. Findings support the operationalization of LHS by translating CMOCs into Action Strategies for those tasked with completing evaluations with a view toward health system learning and innovation sustainability.