Gender bias in resident evaluations: Natural language processing and competency evaluation.

Area of Science:

• Medical Education
• Gender Studies in Medicine
• Natural Language Processing

Background:

• Existing research indicates female trainees may face evaluation penalties for gender non-conforming behavior.
• Studies on gender bias in medical education evaluations show varied results and methodologies.

Purpose of the Study:

• To analyze differences in word use, competency themes, and length in written evaluations of internal medicine residents.
• To investigate the impact of both faculty and resident gender on evaluation feedback.
• To test the hypothesis that female residents receive more negative and different thematic feedback than male residents.

Main Methods:

• Utilized a corpus of 3864 evaluation responses from Yale School of Medicine's internal medicine residency (2012-2018).
• Employed a sentiment model to assess feedback valence and natural language processing (NLP) to analyze feedback content.
• Examined evaluator-evaluatee gender dyads to determine their impact on feedback quantity and quality.

Main Results:

• No substantial differences were found in the number of positive or negative comments received by female versus male residents.
• Gender did not appear to influence the specific Accreditation Council for Graduate Medical Education (ACGME) core competencies discussed in evaluations.
• While the quantity of written feedback did not differ by trainee gender, female evaluators tended to write longer evaluations.

Conclusions:

• Quantitative gender differences in resident evaluations are less prevalent at scale than suggested by qualitative studies.
• Further investigation into linguistic phenomena, such as context, is needed to reconcile these findings with previous research.
• The study highlights the complexity of gender bias in medical education evaluations.