The Use of Artificial Intelligence in Residency Application Evaluation-A Scoping Review

Background Several residency programs have begun investigating artificial intelligence (AI) methods to facilitate application screening processes. However, no unifying guidelines for these methods exist. Objective We sought to perform a scoping review of AI model development and use in residency/fellowship application review, including if bias was explored. Methods A scoping review was performed according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines where a systematic search strategy identified relevant literature within the databases MEDLINE, Embase, and Scopus from inception to September 29, 2023. No limitations on language, article type, or geographic affiliation were placed on the search parameters. Data were extracted from relevant documents, and study findings were synthesized by the authors. Results Twelve studies met inclusion criteria. Most used AI to predict interviews or rank lists (9 of 12, 75%), while the remaining 3 articles (25%) evaluated letters of recommendation with natural language processing. Six articles (50%) compared the model's output to a human-created rank list. Most of the reviewed articles (9 of 12, 75%) mention bias; however, few explicitly modeled biases by accounting for or examining the effect of demographic factors (3 of 12, 25%). Conclusions Few studies have been published on incorporating AI into residency/fellowship selection, and bias remains largely unexplored. There is a need for standardization in bias and fairness reporting within this area of research.