Implementation of standard setting for high-stakes objective structured clinical examinations

Area of Science:

• Medical Education
• Assessment Methods
• Health Professions Education

Background:

• High-stakes Objective Structured Clinical Examinations (OSCEs) traditionally used norm-referenced standards.
• Institutions are exploring criterion-referenced standards for more objective performance evaluation.
• This study addresses the transition from norm-referenced to criterion-referenced standards in OSCEs.

Purpose of the Study:

• To describe an institution's process for transitioning OSCEs from norm-referenced to criterion-referenced standards.
• To evaluate the impact of this standards shift on OSCE performance and pass rates.
• To assess changes in remediation needs following the standards modification.

Main Methods:

• A modified Angoff method was adopted for criterion-referenced standard setting in high-stakes OSCEs.
• OSCE writing teams independently reviewed checklists and determined passing scores.
• Scores, pass rates, and remediation data were compared pre- and post-transition using descriptive statistics.

Main Results:

• Average OSCE scores remained largely unchanged after the transition to criterion-referenced standards.
• The rate of remediation increased significantly, up to 2.6-fold, across multiple years.
• Specific data showed remediation rates rising from 2.8% to 7.4% in year one, and 4.4% to 9% in year four.

Conclusions:

• Transitioning to a criterion-referenced standard setting method (modified Angoff) did not negatively affect average OSCE performance.
• The shift resulted in a notable increase in the identification of students requiring remediation.
• This suggests criterion-referenced standards may offer a more sensitive measure of competency gaps, necessitating proactive support for struggling students.