AI detection in Italian essays through different text representations and adversarial robustness evaluation

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence

Background:

• Distinguishing AI-generated from human-written text is crucial for academic integrity and information authenticity.
• Existing AI detection methods face challenges with evolving AI writing capabilities and adversarial manipulations.

Purpose of the Study:

• To compare the effectiveness of different text representation techniques for AI-generated content detection.
• To assess the robustness of these methods against adversarial attacks.
• To explore the explainability and ethical implications of AI text detection.

Main Methods:

• Utilized a corpus of 1000 Italian essays.
• Employed four text representation methods: Text Features, Most Frequent Words (MFWs), Correspondence Analysis (CA), and fine-tuned Large Language Models (LLMs).
• Applied machine learning classifiers (Random Forests, Elastic-net, Support Vector Machine) to the representations.

Main Results:

• High classification accuracy was achieved across methods, with Text Features initially performing well.
• Models based on Text Features and MFWs were vulnerable to adversarial tactics.
• Correspondence Analysis (CA) and LLM-based approaches demonstrated superior resilience.
• CA offered the best balance of predictor parsimony, accuracy, and robustness to text modification.

Conclusions:

• Correspondence Analysis (CA) and LLM-based methods are promising for robust AI text detection.
• Explainability revealed key linguistic differences between AI and human writing.
• Further research is needed across languages and domains, considering ethical implications in education.