Use of a Machine Learning Program for Urogynecology Fellowship Applicant Review

There is a gap in objective methods to review applications for advanced medical training.

The objective of this study was to evaluate the accuracy of a new machine learning-based residency and fellowship applicant review program, Halsted (Medicratic) in urogynecology fellowship applicant selection compared with program director (PD) review.

This Institutional Review Board-approved study compared PD's standard assessment of fellowship applicants to the Halsted-based assessment at 3 programs in the 2023-2024 application cycle. Each program provided a score for each candidate on a 100-point scale in several domains. After the conclusion of the match, each PD completed a profile within Halsted that identified their preferred qualities in applicants. Halsted scores were obtained, which were compared with PD scores.

A total of 126 applications were reviewed, with 59 applicants reviewed by more than 1 program. Program 1 ( r =0.60; P =0.0019) and Program 2 ( r =0.58; P <0.001) exhibited a significantly strong positive correlation between PD-assigned overall application scores and Halsted scores, while Program 3 exhibited a weak positive correlation between scores ( r =0.33; P =0.0225). There were significant differences in the scoring of the same applicant between programs for PD-assigned mean overall scores ( P <0.001) and Halsted scores ( P <0.001).

A significant positive correlation was found between Halsted rankings of applicants and rankings assigned by PDs. In addition, significant differences in interprogram rankings of applicants suggest that there is a range of qualities that each program values and that application review is individualized. Machine learning assistance in application review is a developing tool with the potential to reduce bias and decrease program administrative burden.