Predictive validity of a tool to resolve borderline grades in OSCEs

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Standard setting methods for examinations lack conclusive evidence regarding validity.
• Examiner uncertainty in high-stakes pass/fail decisions requires effective resolution tools.
• The Objective Borderline Method 2 (OBM2) is designed to reclassify borderline grades.

Purpose of the Study:

• To evaluate the predictive validity of OBM2 pass/fail decisions.
• To assess OBM2's effectiveness in resolving examiner uncertainty in Objective Structured Clinical Examinations (OSCEs).

Main Methods:

• Utilized consecutive years' OSCE results from a medical cohort (n=271) at the University of New South Wales.
• Compared OBM2 decisions from one OSCE (n=687) with subsequent OSCE marks using independent samples T-tests and ANOVA.
• Assessed predictive validity by examining the relationship between OBM2 decisions and future performance.

Main Results:

• OBM2 decisions demonstrated a statistically significant predictive nature for subsequent OSCE marks (p=.005).
• Students whose initial decisions were reclassified to a pass grade achieved significantly higher subsequent OSCE marks than those reclassified to a fail grade.
• Stronger associations were found between related assessment domains (Cohen's d=.469) compared to unrelated domains (Cohen's d=.388).

Conclusions:

• OBM2 decisions exhibit predictive validity, supporting its use in high-stakes OSCEs.
• The method effectively aids in resolving examiner uncertainty for pass/fail decisions.
• OBM2 shows promise as a reliable tool for medical education assessment.