External Validity for Complex Interventions: Towards Pragmatic Strategies for Evaluators.

Area of Science:

• Evaluation Science
• Policy Analysis
• Socio-Ecological Systems

Background:

• Generalizability of evaluation results is crucial for applying findings to new contexts.
• Traditional methods focusing on research design may not suffice for complex interventions.
• Uncovering causal patterns and their applicability in uncertain environments pose significant challenges.

Purpose of the Study:

• To propose pragmatic strategies for addressing generalizability in complex evaluations.
• To guide the selection of appropriate methods and user involvement for generalization.
• To enhance policy learning and stakeholder capacity in navigating complex situations.

Main Methods:

• Development of pragmatic strategies for generalization.
• Inclusion of key questions regarding the scope, assumptions, and methods for generalization.
• Involvement of potential users in the generalization process.
• Case study illustration using evaluations of socio-ecological transition initiatives in France (2013-2020).

Main Results:

• The proposed strategies address critical aspects of generalizability, including purpose, underlying assumptions, and methodological choices.
• User involvement is identified as a key component for successful generalization.
• The case study demonstrates the practical application of these strategies in a real-world complex evaluation.

Conclusions:

• The pragmatic strategies enhance the generalizability of complex evaluation findings.
• These strategies improve policy learning and equip stakeholders to manage uncertainty.
• Effective generalization requires careful consideration of context, assumptions, and stakeholder engagement.