Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School

Area of Science:

• Medical Education
• Clinical Skills Assessment
• Standard Setting Procedures

Background:

• Objective Structured Clinical Examinations (OSCE) are standard for assessing medical students' clinical skills.
• Traditional pass/fail cut-offs in OSCEs show significant variability between student cohorts.
• Standard setting procedures are increasingly emphasized to improve OSCE reliability.

Purpose of the Study:

• To compare the utility, feasibility, and appropriateness of four standard setting methods in OSCEs.
• To evaluate different standard setting approaches at XUSOM.

Main Methods:

• A 15-station OSCE was conducted with 173 students across Respiratory, Gastrointestinal, and Cardiovascular systems.
• Four distinct standard setting methods were applied and compared against a traditional control method.
• Cut-off scores for pass/fail decisions were established using these methods.

Main Results:

• OSCE checklist scores demonstrated acceptable internal consistency (Cronbach's alpha = 0.711).
• The traditional method resulted in the highest failure rate (70 cut-off).
• Modified Angoff and Relative methods showed the lowest failure rates (<10%), while Borderline methods ranged from 28% to 57%.

Conclusions:

• Modified Angoff and Borderline regression methods proved reliable and suitable for setting acceptable OSCE cut-off scores.
• Averaging scores from Modified Angoff and Borderline Regression methods provides a practical approach for OSCE standard setting.
• Further research with more judges and stations is recommended for high-stakes clinical examinations.