Rating the Rater: A Technique for Minimizing Leniency Bias in Residency Applications.

Area of Science:

• Medical Education
• Surgical Training
• Applicant Evaluation

Background:

• Surgical residency programs face challenges in effectively reviewing numerous applicants.
• Faculty ratings of applicants often exhibit significant variability due to leniency bias (Hawk-Dove effect).
• This bias can unfairly influence interview invitations based on rater assignment.

Purpose of the Study:

• To develop and evaluate a technique for minimizing leniency bias in surgical residency applicant scoring.
• To assess the impact of this technique on inter-rater reliability and interview selection.

Main Methods:

• A novel technique was implemented to mitigate leniency bias in scoring 222 plastic surgery residency applicants.
• The variance in applicant ratings by different faculty members was compared before and after applying the technique.

Main Results:

• Median variance in applicant ratings decreased from 0.68 to 0.18 after the technique was applied, indicating improved rater agreement.
• The technique influenced interview invitations for 16 applicants (36% of those interviewed).
• One applicant who ultimately matched would not have been invited for an interview without the bias correction.

Conclusions:

• A simple and effective technique can successfully minimize leniency bias among raters of residency applicants.
• The presented method and resources (instructions, Excel formulae) can be adopted by other programs.
• Implementing this technique promotes more equitable and accurate applicant evaluations in competitive residency matches.