Human or Machine? A Comparative Analysis of Artificial Intelligence-Generated Writing Detection in Personal...

Area of Science:

• Medical Education
• Health Professions Education
• Academic Integrity

Background:

• The rise of advanced AI, including ChatGPT and Google Gemini, presents challenges to the authenticity of academic submissions.
• Previous research indicates variable success rates in detecting AI-generated text, necessitating further investigation.

Purpose of the Study:

• To evaluate the efficacy of human readers, Recurrence Quantification Analysis (RQA), and the GPTZero AI detection tool in differentiating AI-generated from human-written personal statements.
• To assess the implications of these detection methods for physical therapist education program admissions.

Main Methods:

• Fifty human-written and fifty AI-generated personal statements were analyzed.
• Analysis was conducted by two human raters, RQA for lexical sophistication, and GPTZero for AI-specific text characteristics.

Main Results:

• Human raters achieved high agreement (κ = 0.92) and accuracy (97%-99%).
• RQA parameters (recurrence, max line) and GPTZero parameters (simplicity, perplexity, readability) effectively differentiated between human- and AI-generated text, with areas under the ROC curve > 0.768.

Conclusions:

• Human readers, RQA, and GPTZero demonstrate varying but significant accuracy in distinguishing AI-generated from human-written personal statements.
• Findings underscore the importance of these methods for academic admissions and suggest future research on integrated approaches.
• Strategies are proposed for applicants, governing bodies, and institutions to uphold integrity in admissions processes.