Underrepresented in Medicine and International Graduate Bias in the CASPer Situational Judgment Test for...

Situational Judgment Tests (SJTs) are increasingly utilized in medical trainee recruitment decisions yet little is known of their potential biases. This study was designed to evaluate the Computer-Based Assessment for Sampling Personal Characteristics (CASPer) SJT scores of ophthalmology residency applicants and compare these to Step 1 and Step 2 scores in different demographic groups.

All applicants to the ophthalmology residency match completed CASPer during the 2022-2023 recruitment cycle. Applicant scores were cross-referenced within the SF Match Residency Application System to collect data on applicant gender, self-identified Under-Represented in Medicine (URiM) status, and International Medical Graduate (IMG) status, as well as Step 1 and Step 2 scores.

A total of 760 of 765 (99.3%) of applicants with CASPer scores were identified in SF Match, 41% of which were female, 15% URiM, and 11% IMG. Males scored slightly higher on all three tests, with the mean difference in Step 1 (4.3) reaching a statistical significance (Cohen's d = 0.29, p < 0.001). When comparing URiM and IMG applicants, these groups scored significantly lower than their counterparts on all three tests (p all <0.001), with the magnitude of mean difference greater for CASPer than Step 1 or 2. There were no differences between genders in CASPer percentile scores; however, URiM applicants performed on average 17% and IMG applicants 32% lower than their counterparts.

The CASPer SJT demonstrated significant bias against URiM and IMG ophthalmology residency applicants. This bias was even greater than Step 1 and Step 2 performance, suggesting the continued need for caution when utilizing any standardized testing in recruitment decisions.