Variation in performance on common content items at UK medical schools

Area of Science:

• Medical Education
• Health Professions Education Assessment

Background:

• UK medical schools utilize diverse assessment systems, hindering cross-institutional performance comparisons.
• A national licensing assessment is forthcoming, necessitating standardized evaluation methods.
• Evaluating performance on shared knowledge test items can inform cross-school consensus on standard-setting.

Purpose of the Study:

• To compare medical school performances using a common pool of knowledge test items.
• To assess the impact of different standard-setting regimes on candidate pass rates across schools.
• To inform the development of a consensus on standard setting for future licensing assessments.

Main Methods:

• Cross-sectional study conducted over two academic sessions (2016-17, 2017-18).
• Utilized 60 multiple-choice 'common content' knowledge items administered to over 14,000 medical students from 30 UK medical schools.
• Compared school performance using mean scores and modeled the effect of varying passing standards.

Main Results:

• Significant variations in candidate total scores were observed across medical schools (Cohen's d ≈ 1).
• A single passing standard resulted in vastly different fail rates (5% vs. up to 40%) between high- and low-performing schools.
• Hypothetical passing standards demonstrated substantial disparities in pass rates depending on the institution.

Conclusions:

• Candidate performance on common medical knowledge tests differs significantly between institutions.
• Standard-setting for national assessments must account for institutional performance variations to ensure equitable outcomes.
• Further exploration of these institutional differences is essential before implementing shared standards for medical licensing.