Unveiling ChatGPT text using writing style

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• The proliferation of large language models (LLMs) like ChatGPT has led to increased use of AI-generated text.
• Concerns exist regarding the potential for AI text generators to undermine academic integrity and standards.
• Detecting AI-generated content is crucial for maintaining academic honesty.

Purpose of the Study:

• To develop and evaluate a technique for detecting plagiarism in AI-generated texts, particularly those produced by ChatGPT.
• To assess the effectiveness of various machine learning classifiers in distinguishing between human and AI writing styles.
• To propose a superior method for feature extraction compared to traditional techniques like TF-IDF.

Main Methods:

• Extraction and normalization of intrinsic stylometric features from documents.
• Implementation and comparison of classical classifiers (k-Nearest Neighbors, Decision Tree, Naïve Bayes) and ensemble classifiers (XGBoost, Stacking).
• Rigorous evaluation using Cross-Fold validation, hyperparameter tuning, and multiple training iterations.

Main Results:

• Both classical and ensemble learning classifiers demonstrated efficacy in differentiating human and ChatGPT writing styles.
• The XGBoost classifier achieved perfect scores (100%) in accuracy, recall, and precision for detecting AI-generated text.
• The proposed stylometric feature extraction method surpassed TF-IDF techniques, achieving state-of-the-art results on the same dataset.
• Ensemble classifiers correctly identified mixed human-AI texts with 98% accuracy.
• Paragraph-level authorship attribution reached 92.3% accuracy.

Conclusions:

• The proposed stylometric feature extraction and XGBoost classification method is highly effective for detecting ChatGPT-generated plagiarism.
• The approach offers a robust solution for identifying AI-generated content and mixed authorship within documents.
• This technique provides a valuable tool for academic institutions to uphold integrity in the age of AI.