Dubious Evidence, Valuable Information

Area of Science:

• Public Administration
• Policy Analysis
• Program Evaluation

Background:

• Local governments often face decisions about adopting programs demonstrated elsewhere.
• Evaluating the performance of external programs presents significant challenges due to various uncertainties.
• The quality of evidence regarding program success can vary, impacting decision-making.

Purpose of the Study:

• To explore the utility of low-quality evidence in program performance for decision-making.
• To identify and analyze the sources of uncertainty in extrapolating program success from one context to another.
• To provide a framework for local governments considering program adoption.

Main Methods:

• Conceptual analysis of decision-making under uncertainty.
• Identification of five key sources of uncertainty in program extrapolation: chance, confounding variables, component effectiveness, design substitutions, and breakeven criteria.
• Case illustration using a hypothetical local government ('Here') considering a program from 'Elsewhere'.

Main Results:

• Low-quality evidence, despite its limitations, can still be a valuable starting point for decision-making.
• Successful program adoption requires addressing specific uncertainties related to performance assessment, confounding factors, and contextual fit.
• Local governments must conduct their own thorough analysis and information gathering, even when presented with external performance data.

Conclusions:

• Decision-making processes can leverage imperfect evidence by systematically addressing inherent uncertainties.
• Extrapolating program success requires a critical evaluation of external performance data and a clear understanding of local needs and constraints.
• Effective program adoption hinges on a proactive approach to information gathering and risk management.