The 2011 Program Evaluation Standards: a framework for quality in medical education programme evaluations.

Area of Science:

• Program evaluation
• Medical education research
• Quality improvement methodologies

Background:

• The 2011 Program Evaluation Standards were developed from extensive stakeholder input over six years.
• These standards emphasize the interconnectedness of values, meaning, and measurement in program evaluation.
• Evaluation quality is defined by balancing five key attributes: utility, feasibility, propriety, accuracy, and accountability.

Purpose of the Study:

• To apply the 2011 Program Evaluation Standards to a meta-evaluation of a medical undergraduate program.
• To demonstrate how the standards can identify and address inherent tensions, dilemmas, and hazards in evaluation.
• To showcase strategies for designing and conducting high-quality evaluation studies.

Main Methods:

• Utilized the 2011 Program Evaluation Standards as a framework for meta-evaluation.
• Applied the standards to analyze the evaluation of McGill's undergraduate physicianship program.
• Examined the relationships between the five core attributes and the 30 standards, 200 strategies, and 197 hazards.

Main Results:

• The standards effectively illuminated the complexities and challenges encountered throughout the program evaluation process.
• Specific strategies were identified for enhancing the design and execution of evaluation studies.
• The application revealed practical insights into managing tensions and hazards in evaluation.

Conclusions:

• The 2011 Program Evaluation Standards serve as an essential reference for all phases of medical education program evaluations.
• Adherence to the standards promotes more rigorous and effective evaluation practices.
• The meta-evaluation confirmed the practical utility of the standards in real-world educational settings.