Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A...

Area of Science:

• Medical Education
• Health Professions Education Assessment

Background:

• Objective Structured Clinical Examinations (OSCEs) are crucial for assessing clinical skills.
• Establishing valid and reliable cutoff scores for OSCEs is a persistent challenge.
• Various standard-setting methods exist, but they can yield different results.

Purpose of the Study:

• To compare the convergent validity of different standard-setting methods for OSCEs.
• To identify the most reliable method for determining OSCE cutoff scores.
• To evaluate the utility of cluster analysis in validating standard-setting methods.

Main Methods:

• 30 OSCE stations from 393 medical students (Years 2-3) were analyzed.
• Psychometric properties of scores were determined.
• Cutoff scores from Wijnen, Cohen, Mean-1.5SD, Mean-1SD, Angoff, borderline group, and borderline regression (BL-R) methods were compared using ANOVA and Cohen's kappa, alongside three cluster analysis variants.

Main Results:

• The borderline regression (BL-R) and Wijnen methods exhibited the highest convergent validity.
• Angoff and Mean-1.5SD methods showed the least convergent validity.
• Cluster analysis variants demonstrated substantial convergent validity with borderline methods.

Conclusions:

• While the Wijnen method has high convergent validity, it lacks theoretical strength for competency-based assessment.
• The BL-R method is the most valid for OSCEs when compared with other standard-setting approaches.
• Cluster analysis with the mean method can be used for quality assurance of borderline methods, requiring further validation in diverse settings.