The whole is greater than the sum of its parts: Evaluating systems of assessment as complex adaptive systems

Assessment in health professions education has evolved from individual assessment instruments towards integrated systems of assessment that support learning and high-stakes decision-making. While considerable attention has been devoted to the design, validity and reliability of individual assessment methods, far less attention has been paid to evaluating assessment systems themselves. This is an important gap, as assessment systems may contain multiple high-quality assessment tools yet still fail to function as coherent and effective wholes.

This paper proposes a conceptual framework for evaluating systems of assessment in health professions education. Drawing on systems thinking and complexity theory, we argue that contemporary assessment systems, such as programmatic assessment, can be understood as complex adaptive systems and therefore require evaluation approaches that extend beyond the evaluation of individual assessment components.

We trace the evolution of assessment thinking-from individual assessments to systems of assessment-and consider these systems through the lens of complex adaptive systems. We then combine the Logic Model and the Context-Input-Process-Product (CIPP) evaluation framework to connect programme theory with evaluation and examine both intended and unintended consequences.

The integrated framework comprises six components: Planning, Resource Design, Implementation, Short-term Results, Outcomes and Long-term Impact. Together, these components provide a structure for evaluating how assessment systems are designed, implemented, experienced and improved over time. A worked example illustrates how system-level evaluation identifies issues that are not apparent when individual assessment tools are evaluated in isolation.

Evaluating assessment systems requires approaches that examine interactions, implementation processes, educational impact and broader consequences. The proposed framework offers a structured theory-informed approach to support continuous improvement, quality assurance and the development of validity arguments for contemporary systems of assessment. Future research should explore the framework's feasibility, utility and impact across diverse health professions education contexts.