The collective turing test: large language models can generate realistic multi-user discussions

Area of Science:

• Computational Social Science
• Artificial Intelligence Ethics
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for simulating online communities and social media.
• Applications include testing recommendation algorithms and content policies.
• The accuracy of LLM-generated conversations mimicking human interactions is largely unverified.

Purpose of the Study:

• To evaluate the capability of LLMs to convincingly simulate human group conversations on social media.
• To assess the realism of AI-generated content compared to authentic human discussions.

Main Methods:

• Collected authentic human conversations from Reddit.
• Generated artificial conversations on similar topics using Llama 3 70B and GPT-4o.
• Presented both human and AI-generated conversations to participants for evaluation.

Main Results:

• LLM-generated conversations were misidentified as human-created 39% of the time.
• Participants identified Llama 3-generated conversations as AI only 56% of the time, near random chance.
• GPT-4o's performance was not explicitly detailed but implied to be more distinguishable.

Conclusions:

• LLMs can produce social media conversations realistic enough to deceive human readers.
• This capability presents opportunities for social simulation research.
• It also raises concerns about the potential for LLMs to create inauthentic social media content.