Echoes in AI: Quantifying lack of plot diversity in LLM outputs.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Creativity

Background:

• Large language models (LLMs) are increasingly used for creative content generation.
• A key question is whether LLMs can foster genuine collective creativity through diverse ideation.

Purpose of the Study:

• To evaluate the diversity of plot elements generated by state-of-the-art LLMs (GPT-4, LLaMA-3) in story generation.
• To introduce and validate an automatic metric, the Sui Generis score, for measuring plot uniqueness.

Main Methods:

• Examined story generation from GPT-4 and LLaMA-3 using identical prompts.
• Developed the Sui Generis score to quantify the uniqueness of plot elements across multiple LLM generations.
• Conducted human evaluations to correlate Sui Generis scores with perceived surprise levels.

Main Results:

• LLM-generated stories frequently feature echoed plot elements across generations and models.
• Human-written stories exhibit significantly more unique plot elements compared to LLM outputs.
• The Sui Generis score shows moderate correlation with human judgments of surprise, validating its effectiveness.

Conclusions:

• Current LLMs may not sufficiently bolster collective creativity due to repetitive plot element generation.
• The Sui Generis score offers a reliable, automatic method for assessing originality in AI-generated narratives.
• Further research is needed to enhance LLM diversity for creative applications.