Feasibility of Using AI to Evaluate General Surgery Residency Application Personal Statements

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine

Background:

• Graduate medical education increasingly uses artificial intelligence (AI).
• Personal statements (PSs) are subjective yet crucial for residency applications.
• Residency programs are evaluating AI for application screening.

Purpose of the Study:

• Assess the feasibility of using a large language model (LLM) to evaluate general surgery residency PSs.
• Compare AI-generated scores with human-assigned scores for PSs.

Main Methods:

• Retrospective analysis of 668 deidentified general surgery residency PSs.
• Scored PSs using a 1-5 scale in leadership and pathway domains by human assessors (HA) and GPT-3.5 (AI).
• Compared AI and HA scores using descriptive statistics and weighted kappa coefficients; qualitative review of discrepant cases.

Main Results:

• Low agreement between AI and HA scores (κ=0.184 leadership, κ=0.120 pathway).
• AI scored leadership lower (median 3) and pathway higher (median 4) than HA (median 4 leadership, median 3 pathway).
• AI required explicit labels for high scores; HA recognized inferred qualities like resilience and passion.

Conclusions:

• AI demonstrated consistent rubric application but interpreted PSs differently than humans.
• AI may improve consistency and scalability in initial screening.
• Human judgment is vital for evaluating implicit meaning and nuanced content in applications.