Using Kane's Validity Framework to Compare an Integrated and Single-Skill Objective Structured Clinical Examination

The aim of this study was to compare the validity of an integrated objective structured clinical examination (OSCE) station assessing both oral and written components with that of an OSCE station assessing 1 single skill (oral only), both targeted at assessing taking a best possible medication history.

A convergent mixed-methods design that used the 4 inferences of Kane's validity framework (scoring, generalization, extrapolation, and implications) as a scaffold to integrate qualitative data (post-OSCE reflections) and quantitative data (assessment grades and categories of medication errors) was applied.

In 2022, 216 students completed the OSCE station with the oral component alone, while in 2023, 254 students completed the integrated (oral and written) OSCE station. Students in 2023 performed significantly better, with a median score of 88% vs 80% in 2022. There was a greater proportion of commission errors in the integrated assessment (20.4% vs 15.3%), but fewer omission errors (29.9% vs 31.8%) and patient profile errors (5.1% vs 69.4%). Student reflections revealed that conversations were rushed in the integrated assessment, with a greater focus on written formatting, but an appreciation for the authenticity and structured format of the integrated OSCE compared with the single-skill OSCE alone.

Students completing the integrated OSCE (with oral and written components) had fewer patient profile and medication omission errors than students who completed the oral-only OSCE. Considering Kane's validity framework, there was a stronger argument for the more authentic integrated OSCE in terms of the inferences of extrapolation and implications.