Large language models pass a standard three-party Turing test

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Human-Computer Interaction

Background:

• The Turing test is a benchmark for machine intelligence, assessing a machine's ability to exhibit human-like conversation.
• Distinguishing between human and machine intelligence remains a key challenge in AI research.
• Previous evaluations of AI conversational abilities have yielded mixed results.

Purpose of the Study:

• To empirically evaluate the performance of leading large language models (LLMs) in Turing tests.
• To assess whether AI systems can be perceived as human by human evaluators under specific conditions.
• To investigate the criteria humans use when differentiating between humans and AI in conversation.

Main Methods:

• Conducted two randomized, controlled, preregistered Turing tests with independent participant populations.
• Participants engaged in simultaneous 5-minute conversations with a human and an AI system (ELIZA, GPT-4o, LLaMa-3.1-405B, GPT-4.5).
• Evaluated AI performance with and without a "PERSONA" prompt designed to elicit humanlike behavior, and replicated findings in 15-minute games.

Main Results:

• GPT-4.5, when prompted to adopt a human persona, was identified as human 73% of the time, surpassing human performance.
• LLaMa-3.1 achieved a 56% "human" identification rate with the persona prompt.
• Without persona prompts, GPT-4.5 and LLaMa-3.1 performed significantly worse, with identification rates of 38% and 36%, respectively, similar to baseline models.

Conclusions:

• Modern AI systems, particularly when prompted, can empirically pass a standard three-party Turing test.
• Human judgment in these tests relies more on socio-emotional and stylistic factors than traditional intelligence metrics.
• These findings have significant implications for understanding AI capabilities, societal impact, and the uniqueness of human behavior.