Can Residency Programs Detect Artificial Intelligence Use in Personal Statements?

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Academic Integrity

Background:

• The increasing use of AI language models like ChatGPT raises concerns about academic integrity in medical residency applications.
• Personal statements are crucial for residency selection, making the detection of AI-generated content a priority for admissions committees.

Purpose of the Study:

• To assess the effectiveness of widely available AI detection tools in identifying text generated by ChatGPT within personal statements for residency applications.
• To evaluate the reliability of AI detectors for ensuring authenticity in medical education admissions.

Main Methods:

• A qualitative analysis was conducted on 25 personal statements submitted for obstetrics and gynecology residency applications.
• Three AI detection tools—GPTZero, Undetectable AI, and Winston AI—were used to analyze the selected writing samples.

Main Results:

• AI-generated content was highly detectable, while human-written samples showed mixed detection rates (ranging from 3-100%).
• The performance of AI detectors varied, with some tools showing higher accuracy in identifying AI-generated text.
• Human-written statements submitted after the widespread availability of ChatGPT exhibited a broader range of AI detection results.

Conclusions:

• Current AI detection tools may assist residency programs in identifying AI-generated personal statements, but their reliability is not absolute.
• The development of clear guidelines on AI use and authorship is essential to maintain the integrity of application submissions.
• Over-reliance on unvalidated AI detection tools could potentially disadvantage legitimate applicants.