Comparative analysis of text readability and writing styles in AI-generated vs. Human-written academic abstracts

Area of Science:

• Linguistics
• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Research article abstracts are crucial for assessing study significance.
• The rise of AI abstract generation tools (e.g., Kimi, ChatGPT, DeepSeek) prompts evaluation of their output quality.
• Concerns exist regarding the readability and writing style of AI-generated abstracts compared to human-written ones.

Purpose of the Study:

• To compare text readability and writing styles between human-written and AI-generated abstracts.
• To identify differences in abstract quality across linguistics and computer science disciplines.
• To assess the current capabilities and limitations of AI in academic abstract generation.

Main Methods:

• Analysis of 150 human-written and 150 AI-generated abstracts from linguistics and computer science high-impact journals.
• Utilized the Readability Scoring System for quantitative readability and writing style metrics.
• Employed expert evaluation for qualitative assessment of AI-generated academic abstracts.
• Statistical analysis of quantitative data using SPSS 27 with non-parametric methods.

Main Results:

• AI-generated abstracts demonstrated significantly lower readability across eight metrics, indicating increased complexity.
• Discipline-specific analyses revealed five differing metrics in linguistics and eight in computer science.
• Interdisciplinary comparisons showed non-significant differences in nine readability metrics, suggesting AI's mimicry of natural writing.
• AI struggles with generating lexically diverse content, a key aspect of human writing.

Conclusions:

• Current AI models face limitations in producing highly readable and human-like academic abstracts, especially in specialized fields.
• AI shows potential in mimicking natural writing styles but requires further development for lexical richness.
• The findings highlight the need for careful consideration of AI-generated content in scientific publishing.