Setting defensible minimum-stations-passed standards in OSCE-type assessments

Area of Science:

• Medical Education
• Assessment Science
• Psychometrics

Background:

• Objective Structured Clinical Examinations (OSCEs) often use a minimum number of stations passed (MNSP) standard alongside a general cut-score.
• Current MNSP standards are typically fixed in advance with limited justification and do not adapt to variations in test difficulty or administration.
• This lack of adaptability is incongruent with best practices for high-stakes examinations.

Purpose of the Study:

• To investigate and propose empirically validated methods for setting an examinee-centered, criterion-based MNSP standard in OSCEs.
• To develop an MNSP standard that can dynamically adjust based on station and test difficulty.
• To enhance the fairness and defensibility of OSCEs.

Main Methods:

• Utilized mixed modelling on 442 administrations of the PLAB2 exam in the UK.
• Compared four different methods for setting the MNSP standard in an examinee-centered manner.
• Assessed the reliability of the total number of stations passed compared to the total test score.

Main Results:

• The total number of stations passed demonstrated reliability comparable to the total test score (relative g-coefficients of 0.73 and 0.76, respectively).
• Calculating the MNSP based on the predicted number of stations passed by a borderline candidate at the main exam-level cut-score was identified as the preferred method.
• The proposed MNSP standard showed variability across administrations and accounted for approximately 25% of exam-level failures.

Conclusions:

• The proposed examinee-centered approach for setting the MNSP standard is practically feasible and more defensible than fixed, pre-determined standards.
• This method allows the MNSP standard to vary appropriately with test difficulty, enhancing assessment validity.
• Implementing this adaptive MNSP standard can improve the fairness and robustness of OSCEs in various settings.