Competency-based Standard Setting for a High-stakes Objective Structured Clinical Examination (OSCE): Validity...

Area of Science:

• Medical Education
• Assessment and Evaluation
• Clinical Competency

Background:

• Medical educators require robust methods to confirm trainee competency.
• Objective Structured Clinical Examinations (OSCE) are crucial for assessing clinical skills.
• Establishing appropriate pass/fail standards for OSCEs is essential for progression.

Purpose of the Study:

• To develop competency-based pass/fail cut-scores for a graduation-required OSCE.
• To examine the validity evidence supporting these new assessment standards.
• To evaluate the impact of the revised standards on failure rates and remediation.

Main Methods:

• Six clinicians utilized the modified Angoff method to establish cut-scores for an 8-station OSCE.
• Clinicians estimated the probability of minimally competent students answering checklist items correctly.
• Inter-rater reliability, academic achievement differences, educational impact, and response processes were analyzed.

Main Results:

• The new cut-scores resulted in a significantly lower failure rate (5% compared to 29% previously).
• Inter-rater reliability across domains and cases was exceptionally high (.98).
• Pass/fail groups demonstrated significant differences in six of eight academic achievement measures.

Conclusions:

• The modified Angoff method yielded reliable and valid competency-based cut-scores for the OSCE.
• The new standards substantially reduced failure rates and associated remediation burdens.
• Strong validity evidence across multiple metrics supports the use of these revised OSCE standards.