Comparing faculty and artificial intelligence in grading ophthalmology residency applications

The volume of residency applications and data per applicant are increasing with emphasis on holistic review and application inflation. Studies have shown artificial intelligence (AI) could augment human review in resident selection and reveal successful candidates who may otherwise be overlooked. This study determines whether AI successfully predicts match outcomes for ophthalmology residents and could be a valid means of improving the objectivity and efficiency of the residency match process.

This was a prospective study of 642 applicants in the 2023-2024 San Francisco Match cycle. A total of 129 US doctors of ophthalmology and foreign medical graduates were excluded from analysis. The application data of the 513 US medical doctor graduates was studied to predict their match outcomes. Data were received from applicants on September 1, 2023, and were prospectively analyzed by both AI and faculty until the match results were released on February 6, 2024. Faculty utilized a standardized rubric to generate a rank list. In late September 2024, GPT 3.5 Turbo (Azure OpenAI) was given 5 main criteria and no prior examples to prevent bias. Both faculty and AI had access to the entire San Francisco Match application. Main outcome was predictiveness of rank list on match outcome of each applicant.

Both the AI rank list and faculty rank list were predictive of matching to an ophthalmology residency spot (P values < .001). Each 10-percentile increase of the AI ranking had a 23% increase in the odds of a match (odds ratio = 1.23; 95% CI, 1.15-1.32), and each 10-percentile increase of the faculty rank list had a 41% increase in the odds of a match (odds ratio = 1.41; 95% CI, 1.29-1.53).

AI accurately predicts match outcomes and can be used as an adjunct aide to faculty review of applications to reduce the immense administrative workload and human bias.