Detecting Artificial Intelligence-Generated Personal Statements in Professional Physical Therapist Education Program...

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Education
• Academic Admissions

Background:

• The increasing use of AI-generated text, such as ChatGPT, poses challenges to the authenticity of academic applications.
• Personal statements are a key component of admissions for professional programs, including physical therapy.

Purpose of the Study:

• To compare the lexical sophistication of personal statements written by human applicants versus those generated by ChatGPT.
• To evaluate the effectiveness of Recurrence Quantification Analysis (RQA) in distinguishing between human and AI-generated text.

Main Methods:

• Collected 152 human-written and 20 ChatGPT-generated personal statements for physical therapist education programs.
• Analyzed statements using Recurrence Quantification Analysis (RQA) to compare indices like recurrence, determinism, and entropy.
• Employed receiver operating characteristic (ROC) curve analysis to assess RQA's discriminative power.

Main Results:

• ChatGPT-generated statements showed significantly higher recurrence, determinism, mean line, and entropy values compared to human statements.
• A determinism rate of 13.04% was the strongest discriminator, with 70% sensitivity and 91.4% specificity.
• Statements with determinism exceeding 13% were 8 times more likely to be AI-generated.

Conclusions:

• RQA can effectively distinguish AI-generated text from human writing, though not with absolute certainty.
• AI writing tools like ChatGPT present new challenges for admissions committees regarding statement authenticity.
• Admissions committees should consider re-evaluating the role of personal statements and explore tools like RQA for AI detection.