Human Versus Machine Intelligence: Assessing Natural Language Generation Models Through Complex Systems Theory.

Area of Science:

• Complexity Science
• Natural Language Generation (NLG)
• Computational Linguistics

Background:

• Transformer architectures, like GPT-2, excel at Natural Language Generation (NLG), producing human-like text.
• Understanding the underlying information processing of these deep learning models remains a challenge.
• Existing methods for analyzing text complexity are being explored to differentiate machine-generated content.

Purpose of the Study:

• To conduct a comparative analysis of stochastic processes in texts generated by GPT-2, human novels, and programming code.
• To investigate the text-embedding capabilities derived from complexity measures for machine learning applications.
• To enhance the theoretical understanding of deep learning-based NLG systems.

Main Methods:

• Multifractal Detrended Fluctuation Analysis (MF-DFA) and Recurrence Quantification Analysis (RQA).
• Application of Zipf's law and approximate entropy to characterize text properties.
• Development of synthetic text descriptors and feature selection using evolutionary techniques for machine learning.

Main Results:

• GPT-2 generated texts exhibit distinct long-range correlations and recurrence patterns compared to human novels and code.
• Multivariate analysis shows GPT-2 texts positioned between natural language and computer code.
• High accuracy in classifying text types using complexity-derived features indicates their informativeness.

Conclusions:

• Complexity measures provide valuable insights into the statistical properties of NLG outputs.
• The proposed methodology can improve text classification, fake news detection, and plagiarism detection systems.
• This research contributes to understanding the "black-box" nature of deep learning models in NLG.