Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can 'work'

Area of Science:

• Medical Education
• Assessment Methodology
• Psychometrics

Background:

• Borderline regression (BRM) is often deemed unsuitable for small cohort Objective Structured Clinical Examinations (OSCEs), leading institutions to use resource-intensive item-centered standard-setting methods.
• These traditional methods may lack defensibility, particularly in performance-based assessments.

Purpose of the Study:

• To investigate the applicability and robustness of BRM in small-cohort OSCE contexts.
• To challenge the assumption that BRM is problematic for smaller sample sizes in educational assessments.

Main Methods:

• Analysis of post-hoc station- and test-level metrics from three distinct small-cohort OSCEs.
• Contexts included international medical graduate exams, senior undergraduate exams, and Physician Associate exams.

Main Results:

• BRM yielded robust metrics and defensible cut scores in most stations across the studied contexts (5-14% problematic stations).
• Issues arose when the correlation between global grades and checklist scores was insufficient to support the BRM-determined standard.

Conclusions:

• BRM can provide defensible standards in small test cohorts if there is adequate ability spread and careful design of scoring instruments.
• Existing station cut-scores serve as a preferred alternative when BRM encounters standard-setting problems.