Validation of educational assessments: a primer for simulation and beyond

Area of Science:

• Health Professions Education
• Assessment Science
• Educational Measurement

Background:

• Simulation is crucial for health professions assessment.
• This review serves as an introduction to assessment validation for educators and researchers.
• Principles discussed apply broadly beyond simulation-based assessments.

Purpose of the Study:

• To provide a primer on assessment validation for health professions educators and researchers.
• To explain contemporary frameworks for validity evidence collection.
• To outline a practical, eight-step approach to validation.

Main Methods:

• Defining the construct and proposed interpretation.
• Explicitly stating intended decisions and the interpretation-use argument.
• Prioritizing validity evidence, appraising existing evidence, and collecting new evidence as needed.
• Formulating a coherent validity argument based on empirical evaluation.

Main Results:

• Validation involves collecting evidence to support or refute the hypothesis that interpretations and decisions based on assessment results are appropriate.
• Frameworks by Messick and Kane offer different perspectives on validity evidence sources and key inferences.
• An eight-step approach is proposed for systematic validation applicable to various assessment contexts.

Conclusions:

• Rigorous validation requires prioritizing and empirically evaluating assumptions related to assessment score interpretation and use.
• Enhancing validation science involves clearer articulation of the interpretation-use argument and greater use of formal frameworks.
• More evidence is needed regarding the consequences and implications of assessment validation.