Artificial Intelligence in Personal Statements Within Orthopaedic Surgery Residency Applications

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Surgical Residency Applications

Background:

• Artificial intelligence (AI) is increasingly integrated into medical education and clinical practice.
• The use of AI for crafting personal statements (PSs) in orthopaedic surgery residency applications is currently unexamined.
• This study investigates the prevalence of AI-generated content in orthopaedic surgery residency PSs.

Purpose of the Study:

• To analyze orthopaedic surgery residency personal statements (PSs) for the presence of AI-generated text.
• To determine the rate of AI utilization in PSs submitted to an orthopaedic surgery residency program.
• To evaluate the efficacy of AI detection software in distinguishing human-generated from AI-generated text.

Main Methods:

• Comparison of four groups: pre-ChatGPT (PRE-PS), post-ChatGPT (POST-PS), AI-generated (AI-PS), and hybrid (H-PS) personal statements.
• Utilized AI detection software (GPT-Zero) to quantify human, mixed, and AI-generated text percentages.
• Assessed AI detection software's confidence levels for classifying statements as human or AI-generated.

Main Results:

• Human-generated text percentages: PRE-PS (94%), POST-PS (93%), H-PS (28%), AI-PS (0%).
• AI detection software classified all 200 submitted PSs as human with >90% confidence.
• AI-generated statements (H-PS and AI-PS groups) were correctly identified as "AI" with 100% confidence for the AI-PS group.

Conclusions:

• Current orthopaedic surgery residency applicants do not appear to be utilizing AI for personal statement creation.
• AI detection software (GPTZero) demonstrates high accuracy in identifying AI-generated PSs for residency applications.
• Future research should monitor AI utilization trends, and guidelines for AI use in postgraduate applications should be established.