Artificial Intelligence-Assisted Narratives: Analysis of Surgical Residency Personal Statements

Personal statements (PSs) express applicants' personal characteristics and motivations informing pursuit of a surgical career. Generative artificial intelligence (AI) is a revolutionary technology. There are currently no data to suggest how and to what extent AI is used in surgical residency applications. We examined the prevalence of AI use and applicant pool characteristics in PSs submitted to a surgical residency.

PSs from US MD and DO applicants to an academic general surgery program were collected for both the 2022-23 and 2023-24 NRMP Match cycles. PSs were analyzed using 2 AI-detection tools: GPTZero and Copyleaks. Data were analyzed using T-test and Fisher's Exact Test.

UMass Chan Medical School-Baystate general surgery residency program in Springfield, Massachusetts.

There were 1332 applications during 2022-23 NRMP Match cycle and 1221 for 2023-24. After excluding international medical graduates and incomplete applications, 1490 PSs were analyzed.

1490 PS were included (758 [50.9%] for 2022-23; 732 [49.1%] for 2023-24). Demographic characteristics did not differ between the 2 cycles. GPTZero identified AI use in 77 (10.2%) PSs in 2022-23 and 268 (36.6%) in 2023-24 (p < 0.001). Copyleaks identified AI use in 20 (2.6%) PSs in 2022-23 and 165 (22.5%) in 2023-24 (p < 0.001). Concordance in AI detection with both tools was observed in 13 (1.7% of total PSs) for 2022-23 and 155 (21.2%) for 2023-24 (p < 0.001). Subgroup analysis of concordance in 2023-24 showed increased non-English native language characteristics (38.7% vs 19.6%; p < 0.001), a lower average personal statement word count (597.3 vs 645.9; p < 0.001) and shorter average sentence (10.0 vs 10.4 words; p < 0.001) in the AI group.

Although AI-detection tools are imperfect, demonstration of increased AI use in personal statement preparation is compelling. Implications of AI use in residency applications are unknown, and programs must develop policies anticipating ongoing and potentially increased use of AI in the upcoming application cycles.