To prove or improve? Examining how paradoxical tensions shape evaluation practices in accreditation contexts.

Area of Science:

• Health Professions Education
• Program Evaluation
• Academic Health Sciences

Background:

• Programme evaluation is increasingly routinised in academic health sciences.
• There is limited research on factors influencing evaluation scope and quality in health professions education.
• This study examines how the practice context shapes evaluation in health professions education.

Purpose of the Study:

• To investigate how the context of accreditation influences evaluation practices in academic health sciences.
• To identify and critically examine paradoxical tensions arising from accreditation demands versus evaluation ideals.
• To propose strategies for optimizing evaluation quality within accreditation-seeking contexts.

Main Methods:

• Qualitative research within a critical realist paradigm.
• Semi-structured interviews with 29 individuals overseeing evaluation work across 26 institutions in 4 regions.
• Framework and matrix analyses of collected data.

Main Results:

• Three overarching themes emerged: absence of collective coherence, disempowerment of expertise, and tensions in routine practice.
• Identified tensions include: resourcing accreditation vs. robust strategy, accreditation design vs. renewal, and public dissemination vs. restricted access.
• These tensions risk the quality and credibility of evaluation practices.

Conclusions:

• The high-stakes context of accreditation surfaces tensions that can compromise evaluation quality and credibility.
• Commissioners and executors of evaluation must identify and reconcile these tensions to mitigate risks.
• Evaluation should be viewed as a socio-technical practice vulnerable to context, not merely a method.