Gotcha GPT: Ensuring the Integrity in Academic Writing.

Area of Science:

• Computer Science
• Natural Language Processing
• Academic Publishing

Background:

• The rapid advancement of Artificial Intelligence (AI) presents challenges to ensuring the integrity of academic writing.
• Distinguishing between AI-generated and human-generated manuscripts is crucial for universities and publishers.
• Existing methods may not be sufficient to address AI's growing capabilities in text generation.

Purpose of the Study:

• To develop and evaluate machine learning models for differentiating AI-generated from human-generated academic text.
• To provide a reliable tool for academics and publishers to verify manuscript authorship.
• To offer practical solutions for maintaining academic integrity in the age of AI.

Main Methods:

• Utilized classifier models including decision tree, random forest, extra trees, and AdaBoost.
• Employed Scikit learn libraries for statistical evaluation (precision, accuracy, recall, F1, MCC, Cohen's kappa) and confusion matrix analysis.
• Trained and tested models on a dataset of approximately 400 AI-generated and 400 human-generated scientific manuscript texts with a 50/50 random split.

Main Results:

• Model evaluation accuracy for classification ranged from 0.97 to 0.99.
• The employed statistical metrics and confusion matrix provided high confidence in the model's performance.
• The models demonstrated a strong capability to distinguish between AI and human-generated text.

Conclusions:

• The developed classifier models are effective in identifying AI-generated academic writing with high accuracy.
• This approach offers a valuable tool for safeguarding academic integrity in scholarly publications.
• Freely available tutorials and code (Gotcha GPT) support the practical application of these methods.