Increased Prevalence of Large Language Model-Associated Language in US Residency Application Personal Statements

Residency leaders increasingly rely on personal statements to assess applicant communication skills and individuality. Personal statements written by large language models (LLMs) have been rated more readable but more generic than applicant-written ones, yet the prevalence of LLM use and its effect on applicant writing remain unknown.

To measure the prevalence of LLM-associated language in personal statements from the 2017 through 2024 application cycles and assess associated changes in applicant writing.

This retrospective descriptive study analyzed personal statements submitted to the internal medicine, family medicine, pediatrics, emergency medicine, and general surgery residencies of a US academic medical center for the 2017 through 2024 application cycles. Using one-sample t tests, prevalence in 2024 of 27 LLM-associated words and 12 control words, identified in prior research, was compared to 2017-2023, the period before widespread LLM release. Secondary outcomes assessing writing characteristics reflective of communication skills and individuality included linguistic complexity (Flesch Reading Ease, word count), lexical diversity (type-token ratio), personalization (first- and third-person pronoun frequency), and topic entropy.

Among 62 132 statements (55 199 from 2017-2023, 6933 from 2024), LLM-associated word prevalence increased from 27.9% (SD, 1.05%) pre-2024 to 45.6% in 2024 (P<.001). Average length decreased from 717.2 (SD, 13.6) words to 676.6 (P<.001), while the type-token ratio increased from 0.49 (SD, 0.006) to 0.51 (P<.001). Approximately 3 fewer first-person pronouns were used per statement (P<.001). Topic entropy remained stable.

LLM-associated language increased 17.7% in 2024, corresponding to approximately 1 in 6 additional personal statements, while writing characteristics reflective of communication skills and individuality changed modestly.