Detecting "large language models fingerprint" for Japanese texts generated by six LLMs

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• The rapid advancement of large language models (LLMs) necessitates methods for distinguishing AI-generated content.
• Identifying unique linguistic characteristics, or "fingerprints," is crucial for differentiating texts produced by various LLMs.

Purpose of the Study:

• To investigate the effectiveness of stylometric features in distinguishing Japanese texts generated by six distinct LLMs.
• To explore whether LLM-generated texts exhibit unique, identifiable linguistic patterns.

Main Methods:

• Utilized Uniform Manifold Approximation and Projection (UMAP) for visual exploration of text distributions.
• Employed Random Forest (RF) and XGBoost with leave-one-out cross-validation for LLM differentiation.
• Applied SHapley Additive exPlanations (SHAP) to identify key stylometric features, including function-word unigrams, part-of-speech (POS) bigrams, and phrase patterns.

Main Results:

• UMAP showed distinct text clusters for most LLMs, with Llama 3.1 exhibiting overlap.
• Random Forest achieved high performance (macro F1 > 0.95, reaching 1.00 for some LLMs) in differentiating texts.
• XGBoost demonstrated strong but slightly lower performance (macro F1 0.88-0.94), with SHAP highlighting specific LLM-dependent feature patterns.

Conclusions:

• Stylometric features, particularly their combinations and patterns, can accurately distinguish Japanese texts generated by different LLMs.
• These findings suggest the existence of LLM-specific linguistic fingerprints, independent of shared underlying transformer architectures.