Applying Kane's validity framework to a simulation based assessment of clinical competence

Area of Science:

• Medical Education
• Assessment Science

Background:

• Clinical competence assessment is complex and requires robust validity evidence, especially for high-stakes decisions.
• Simulation-based assessment is increasingly used, necessitating rigorous evaluation of its validity.

Purpose of the Study:

• To organize, collect, and interpret validity evidence for a high-stakes simulation-based assessment strategy for paramedic certification.
• To apply Kane's validity framework to evaluate the assessment strategy and provide a practical example for others.

Main Methods:

• Utilized Kane's validity framework, examining four inferences: scoring, generalization, extrapolation, and implications.
• Collected and analyzed data related to tool characteristics, reliability, sampling strategies, and correlations with clinical practice.

Main Results:

• Evidence supported the assessment's validity for scoring, generalization, and implications (low error rates).
• Weak evidence was found for extrapolation, as assessment scores did not strongly correlate with clinical errors in practice.

Conclusions:

• The simulation-based paramedic assessment strategy is supported for certification based on scoring, generalization, and implications.
• Further refinement is needed to strengthen the extrapolation evidence, linking assessment performance to actual clinical error rates.