Words matter: using natural language processing to predict neurosurgical residency match outcomes

Area of Science:

• Neurosurgery Residency Admissions
• Medical Education Research

Background:

• Narrative letters of recommendation (NLORs) are crucial for neurosurgery residency applications.
• Their predictive utility for match outcomes versus objective data is not well-established.

Purpose of the Study:

• To compare machine learning models using NLORs and demographics to predict neurosurgery residency match outcomes.
• To determine if narrative language in NLORs predicts standardized letter of recommendation (SLOR) rankings.

Main Methods:

• Analyzed 1498 NLORs from 391 applications (2020-2021 neurosurgery cycle).
• Used logistic regression with LASSO to predict match outcomes and SLOR rankings from NLOR text and demographics.
• Model performance assessed using area under the curve (AUC).

Main Results:

• NLOR and demographics models showed similar discrimination for match outcomes (AUCs 0.75 and 0.80).
• Keywords like "outstanding" and "AOA" predicted match success.
• NLOR language predicted top 5% SLOR rankings, with words like "highest" and "best" being significant.

Conclusions:

• Both NLORs and demographics effectively predict neurosurgery residency match success.
• NLORs offer unique insights into applicant suitability and predict SLOR rankings.
• Narrative evaluations remain valuable components of the residency selection process.