Examining Differential Rater Functioning and Bias in the Holistic Review of Residency Applications

The impact of rater bias in applicant selection for residency interviews using holistic review is poorly understood.

To measure faculty raters' bias toward applicant gender and underrepresented in medicine (URiM) status in their recommendations to offer residency interviews.

In this cross-sectional study, applicant (n=3875) and faculty rater (n=69) data within a single institution's dermatology residency application database from 2014 to 2023 were examined for differences in rater judgments to interview applicants as measured by logits (ie, a standardized unit of scoring difference) using the many-facet Rasch model, a statistical method that accounts for variations among raters, applicants, and scoring for applicant gender (23% women, 14% men, and 63% unknown) and URiM status (26% White, 21% non-White non-URiM, 12% URiM, 41% unknown).

Rater bias, as measured in logits using the rater severity measure, was minimal for applicant gender (-0.12, 0.08 logits) and URiM status (-0.07, 0.04 logits). However, raters were more lenient (ie, likely to favor an interview) when the applicant's and rater's gender (-1.74 logits, SE 0.05) or URiM status (-0.46 logits, SE 0.05) were concordant. Conversely, raters were more severe (ie, unlikely to favor an interview) when discordant on gender (1.74 logits, SE 0.05) or URiM status (0.46 logits, SE 0.05). More than 50% of individual raters exhibited substantial gender bias, and approximately 33% for URiM status bias when concordant.

Raters were more likely to favor offering interview invitations to applicants with concordant gender or URiM status and less likely to favor interviewing applicants with discordant gender or URiM status.