Detecting Artificial Intelligence-Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled...

Area of Science:

• Artificial Intelligence (AI) in Academia
• Scientific Writing Authenticity
• Medical Education Technology

Background:

• Sophistication of AI models like ChatGPT challenges human vs. AI text differentiation.
• Academic integrity concerns are rising, especially in medicine, due to AI text generation.
• Accurate authorship attribution is critical for scholarly work.

Purpose of the Study:

• To assess medical professionals' and humanities scholars' ability to distinguish AI-generated from human-written German medical texts.
• To analyze the reasoning behind expert identification, focusing on content familiarity and linguistic cues.
• To evaluate the impact of specific linguistic features on authorship identification.

Main Methods:

• A semirandomized controlled study involving 35 experts (22 medical, 13 humanities) evaluating paired medical texts.
• Experts identified AI-generated (ChatGPT 3.5) vs. student-written texts and provided justifications.
• Qualitative analysis of justifications and quantitative analysis of text characteristics and expert ratings.

Main Results:

• Experts accurately identified AI-generated texts in 70% of cases, with minimal difference between medical and humanities groups.
• Identification accuracy was largely independent of content familiarity.
• Stylistic features, including redundancy (OR 6.90), repetition (OR 8.05), and coherence (OR 6.62), were crucial for AI text detection.

Conclusions:

• Medical and humanities experts can identify AI-generated texts based on linguistic attributes, not content expertise.
• Redundancy, repetition, and coherence are key indicators of AI authorship in scientific writing.
• Further research should explore training methods based on these linguistic features to improve AI text detection.