Evaluating large language models in theory of mind tasks

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Developmental Psychology

Background:

• Theory of Mind (ToM) is crucial for human social interaction.
• LLMs are increasingly sophisticated AI systems.
• Assessing ToM in AI is a novel research area.

Purpose of the Study:

• To evaluate the Theory of Mind (ToM) capabilities of large language models (LLMs).
• To compare LLM performance on ToM tasks with human developmental benchmarks.
• To explore the implications of emergent ToM-like abilities in AI.

Main Methods:

• Eleven LLMs were tested on 40 bespoke false-belief tasks.
• Tasks included false-belief, true-belief, and reversed scenarios.
• Performance was measured by the number of tasks correctly solved across all eight scenario types.

Main Results:

• Older LLMs failed all tasks.
• GPT-3 and ChatGPT-3.5-turbo achieved 20% accuracy.
• ChatGPT-4 reached 75% accuracy, comparable to 6-year-old children.

Conclusions:

• LLMs demonstrate emerging ToM-like abilities.
• These abilities may be an emergent property of advanced language skills in AI.
• The development of socially skilled AI has significant implications.