Using Natural Language Processing and Machine Learning to Identify Internal Medicine-Pediatrics Residency Values in...

Area of Science:

• Medical Education
• Computational Medicine
• Health Informatics

Background:

• Holistic review in residency admissions is effective but time-consuming.
• Traditional screening relies on quantitative metrics, risking socioeconomic and racial bias.
• Unstructured application data requires extensive resources for evaluation.

Purpose of the Study:

• To develop and evaluate a machine learning model (MLM) for screening residency applicants.
• To assess the MLM's ability to identify key values associated with resident success from application narratives.
• To compare MLM performance against manual holistic review.

Main Methods:

• Extracted text snippets from narrative sections of internal medicine-pediatrics residency applications (2015-2019).
• Expert reviewers annotated snippets for 10 predefined values (e.g., academic strength, compassion, teamwork).
• Prospectively applied the trained MLM to 2023 applications and compared results with manual holistic review.

Main Results:

• The MLM demonstrated moderate sensitivity (0.64) and high specificity (0.97).
• Key application metrics derived from the MLM significantly correlated with interview invitations (P < .001).
• Eight of ten assessed values were significant predictors of interview status.

Conclusions:

• An MLM can effectively identify important values for resident success in internal medicine-pediatrics programs.
• The developed MLM offers a potentially more efficient and objective screening tool.
• Future work includes refining the MLM through increased annotations and parameter tuning.