Artificial Intelligence-Assisted Narratives: Analysis of Surgical Residency Personal Statements

Area of Science:

• Medical Education
• Surgical Residency Admissions
• Artificial Intelligence in Healthcare

Background:

• Personal statements (PSs) are crucial for evaluating surgical residency applicants' motivations and characteristics.
• Generative artificial intelligence (AI) presents a transformative technology with unknown implications for academic applications.
• Current data on AI utilization in surgical residency applications are lacking.

Purpose of the Study:

• To investigate the prevalence of AI use in personal statements submitted for surgical residency applications.
• To analyze applicant pool characteristics associated with AI-detected personal statements.

Main Methods:

• Personal statements from US MD and DO applicants to a general surgery residency program were analyzed for the 2022-23 and 2023-24 NRMP Match cycles.
• Two AI-detection tools, GPTZero and Copyleaks, were employed to identify AI-generated content.
• Statistical analysis included T-tests and Fisher's Exact Tests to compare data between cycles and subgroups.

Main Results:

• AI detection in personal statements rose substantially from 10.2% (GPTZero) and 2.6% (Copyleaks) in 2022-23 to 36.6% (GPTZero) and 22.5% (Copyleaks) in 2023-24.
• Concordant AI detection across both tools increased from 1.7% in 2022-23 to 21.2% in 2023-24 (p < 0.001).
• In 2023-24, AI-assisted statements were more common among non-native English speakers and had lower word counts and shorter average sentence lengths compared to human-written statements.

Conclusions:

• The increasing use of AI in preparing personal statements for surgical residency applications is evident, despite limitations in current detection tools.
• The implications of AI use in residency admissions remain unclear.
• Surgical residency programs must proactively establish policies to manage the anticipated rise in AI utilization during application cycles.