Using Kane's Validity Framework to Compare an Integrated and Single-Skill Objective Structured Clinical Examination

Area of Science:

• Medical Education
• Assessment Methods
• Pharmacy Practice

Background:

• Objective Structured Clinical Examinations (OSCEs) are crucial for evaluating clinical skills.
• Assessing medication history taking is vital for patient safety.
• Traditional OSCE stations often focus on single skills, potentially limiting comprehensive evaluation.

Purpose of the Study:

• To compare the validity of an integrated OSCE station (oral and written components) with an oral-only OSCE station for assessing medication history taking.
• To evaluate the impact of integrated assessment on medication error types and student performance.

Main Methods:

• Convergent mixed-methods design utilizing Kane's validity framework.
• Integration of qualitative (student reflections) and quantitative data (assessment grades, medication error categories).
• Comparison of student performance and error types between an oral-only OSCE (2022) and an integrated OSCE (2023).

Main Results:

• Students in the integrated OSCE (2023) achieved higher median scores (88%) than in the oral-only OSCE (2022) (80%).
• The integrated assessment showed a higher proportion of commission errors but fewer omission and patient profile errors.
• Student reflections indicated appreciation for the integrated OSCE's authenticity and structure, despite some concerns about rushed conversations and written formatting.

Conclusions:

• The integrated OSCE with oral and written components demonstrated superior validity in assessing medication history taking.
• Fewer patient profile and medication omission errors were observed in the integrated OSCE.
• Kane's validity framework supported the integrated OSCE, particularly regarding extrapolation and implications inferences.