Machine translationese of large language models: Dependency triplets, text classification, and SHAP analysis.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Distinguishing human translations from machine translations is crucial for quality assessment.
• Large Language Models (LLMs) generate translations that can be difficult to differentiate from human work.
• Understanding the linguistic characteristics of LLM-generated text (translationese) is an ongoing challenge.

Purpose of the Study:

• To develop and evaluate methods for distinguishing human translations from LLM-generated translations.
• To identify key linguistic features that differentiate human and machine translation.
• To assess the performance of various machine learning classifiers in this task.

Main Methods:

• Utilized dependency triplet features extracted from translation datasets.
• Evaluated 16 different machine learning classifiers.
• Employed 10-fold cross-validation for robust performance assessment.
• Applied SHAP (SHapley Additive exPlanations) analysis to interpret model decisions.

Main Results:

• The Support Vector Machine (SVM) model achieved the highest mean F1-score of 93%.
• All evaluated classifiers demonstrated consistent ability to differentiate between human and machine translations.
• SHAP analysis identified specific dependency features crucial for distinguishing translation origins.

Conclusions:

• Dependency triplet features are effective in distinguishing human from LLM translations.
• Machine learning classifiers, particularly SVM, show high performance in this classification task.
• The study enhances understanding of LLM translationese and informs improvements in translation quality assessment and model development.