Can borderline regression method be used to standard set OSCEs in small cohorts?

Absolute methods of standard setting (SS) are more suitable for OSCEs. However, previous studies have theorised that borderline regression method (BRM) is not reliable for small sample sizes.

OSCE results for the 2017-2019 cohorts were analysed to compare BRM versus Modified Angoff. We reported on whether the stations in multiple cohorts were sufficiently equivalent to aggregate the results together to calculate which method of SS was more reliable. We finally used the bootstrapping method to compare the accuracy of BRM for small versus simulated larger cohorts.

BRM was a valid method for SS OSCEs in this dataset when station quality was sufficiently high. However, a large gap between the Angoff and BRM in some of the OSCEs was observed, which could be explained by poor use of the grading scale. Model fit statistics were generally adequate even with low sample sizes. Using the bootstrap of datasets, the error rate was much higher for low-quality stations but was not an issue in high-quality ones.

This study adds to the evidence that well-designed OSCEs can use BRM for small cohorts. However, there is a need for the institutions to properly assess their stations and their assessors, before embarking into using this method, to prevent from having to remove stations.

This analysis suggests that BRM is an acceptable replacement for Angoff SS in small cohorts, where there is a range of candidates undertaking the assessments and there are well-designed OSCES with well-trained examiners.