Generative AI in Otolaryngology Residency Personal Statement Writing: A Mixed-Methods Analysis

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Residency Admissions

Background:

• Generative Artificial Intelligence (GAI) is increasingly accessible for drafting personal statements.
• This poses challenges for evaluating genuine applicant writing ability and personal insight in residency applications.

Purpose of the Study:

• To compare the quality of GAI-generated personal statements against those written by successful OHNS residency applicants.
• To analyze evaluator perceptions using statistical and qualitative thematic methods.

Main Methods:

• Collected personal statements from successful OHNS residency applicants.
• Generated GAI statements using ChatGPT 4.0 based on applicant statement characteristics.
• Blindly reviewed all statements by 21 experienced evaluators on authenticity, readability, personability, and overall quality.
• Conducted quantitative analysis with independent t-tests and qualitative thematic analysis using NVivo.

Main Results:

• GAI statements significantly outperformed applicant statements in authenticity (7.67 vs. 7.05), readability (8.03 vs. 7.49), personability (7.33 vs. 6.72), and overall score (7.49 vs. 6.90).
• Thematic analysis indicated GAI statements were perceived as "well-constructed but generic," while applicant statements were "verbose and lacked focus."
• Reviewers expressed concerns regarding personal insight and engagement in GAI-generated statements.

Conclusions:

• GAI-generated personal statements received higher ratings, prompting reevaluation of their role in residency applications.
• The findings highlight the need for clear ethical guidelines regarding AI use in medical education and residency admissions.