Feasibility of Using AI to Evaluate General Surgery Residency Application Personal Statements

As artificial intelligence (AI) becomes increasingly integrated into graduate medical education, residency programs are exploring AI's role in application screening. Personal statements (PSs) remain a highly subjective yet influential component of the residency application. This study assesses the feasibility of using a large language model (LLM) to evaluate general surgery residency PSs compared to human-assigned scores.

We conducted a retrospective analysis of 668 deidentified PSs submitted to our general surgery residency program during the 2023-2024 application cycle. PSs were originally scored by human assessors (HA) using an anchored 1-5 scale in two domains: leadership and pathway. Each PS was subsequently scored by GPT-3.5 (AI) using the same rubric and standardized prompts. Descriptive statistics were used to compare AI and HA scores. Inter-rater agreement was assessed using weighted kappa coefficients. Discrepant cases (score differences >2 points) were reviewed qualitatively to identify scoring themes.

AI and HA scoring showed low agreement: κ = 0.184 for leadership and κ = 0.120 for pathway domains. Median AI leadership scores were lower (3 [IQR 2-4]) than HA scores (4 [IQR 3-5]), while AI pathway scores were higher (4 [IQR 4-5]) than HA scores (3 [IQR 3-4]). Qualitative review revealed that AI required explicit labeling (e.g., formal leadership titles or stated adversity) to assign higher scores, whereas HA rewarded inferred qualities such as resilience, passion, and longitudinal commitment.

AI applied rubric-based scoring consistently but interpreted narrative content differently than human reviewers. While AI may enhance consistency and scalability in early application screening, its limitations in recognizing implicit meaning suggest human judgment remains essential for evaluating nuanced or inferential content. Caution should be exercised in adopting AI tools for subjective application review.