Systematic iteration between model and methodology: A proposed approach to evaluating unintended consequences

Area of Science:

• Evaluation Science
• Systems Thinking
• Program Management

Background:

• Logic models are crucial in evaluations but require continuous revision with empirical data for maximum utility.
• The value and limitations of models in scientific endeavors are discussed, setting the context for evaluation methodology.

Purpose of the Study:

• To advocate for systematic integration of empirical data collection and model building in evaluations.
• To enhance evaluators' ability to manage unintended consequences through improved methodologies.
• To explore factors influencing model development and revision in evaluation.

Main Methods:

• The article discusses the relevance of complex systems behavior and generic patterns of change for understanding unintended consequences.
• It examines social, organizational, and cognitive factors affecting program design and outcome identification.
• A process for systematic iteration between model and methodology is outlined.

Main Results:

• Systematic iteration between model building and empirical data collection can improve the timely operationalization of methodological changes.
• Understanding complex systems, generic change patterns, and cognitive biases is key to addressing unintended consequences.
• The embedded nature of programs in fluctuating settings complicates change discernment.

Conclusions:

• Evaluators can better manage unintended consequences by systematically iterating between model development and empirical data collection.
• Further research is needed to optimize the efficiency and effectiveness of the model-data iteration process in evaluations.
• Appreciating complex system dynamics and cognitive influences is vital for robust evaluation design.