AI detection in Italian essays through different text representations and adversarial robustness evaluation

This study investigates the effectiveness of various text representation methods in distinguishing between AI-generated and human-written content, using a corpus of 1000 Italian essays. Four techniques were employed for text representation: Text Features, Most Frequent Words (MFWs), Correspondence Analysis (CA), and a fine-tuned Large Language Model (LLM). Machine learning models, including Random Forests, Elastic-net, and Support Vector Machine, were applied to these representations. The study achieved high classification accuracy, with Text Features performing exceptionally well. However, adversarial tactics revealed vulnerabilities in models based on Text Features and MFWs, while approaches based on CA and LLM demonstrated greater resilience. CA in particular proved to be the best compromise between parsimony in the number of predictors, accuracy and robustness to targeted text-modification attacks. The research also focused on the explainability of the results, highlighting significant differences between AI- and human-written texts, including sentence structure and lexical choices. Ethical considerations, particularly in educational settings, were emphasized, along with the need for further cross-linguistic studies and diverse domain datasets. This research provides valuable insights into the challenges and potential solutions in AI text detection, emphasising the importance of balancing accuracy with interpretability and robustness against adversarial attacks.