Using Kane's validity framework to examine the implications of feedback in simulation-based assessments.

Area of Science:

• Medical Education
• Simulation-Based Assessment
• Validity Evidence

Background:

• Simulation is increasingly used in programmatic assessment, but validity evidence is often lacking.
• Kane's validity framework provides a structured approach to appraising assessment validity.
• The echocardiography competence assessment tool (ECAT) was developed for simulation-based learning.

Purpose of the Study:

• To apply Kane's validity framework to the ECAT.
• To evaluate the ECAT's utility as an assessment for learning.
• To focus on the implications of the ECAT for promoting trainee learning.

Main Methods:

• Implemented the ECAT with 14 cardiology trainees and 4 raters.
• Collected simulation performance data and conducted post-assessment interviews.
• Used directed content and quantitative analyses to generate validity evidence.

Main Results:

• ECAT scores accurately reflected trainee performance and identified learning opportunities.
• High inter-rater reliability (ICC = 0.913).
• ECAT scores correlated with cardiology exam scores and positively associated with diagnostic quality and experience.

Conclusions:

• A validity argument supports the ECAT as a simulation-based assessment for learning.
• Findings suggest areas for optimizing the ECAT.
• The study illustrates Kane's framework for educators using simulation-based assessments.