Testing theory of mind in large language models and humans

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Linguistics

Background:

• Theory of Mind (ToM) is a key human cognitive ability, involving tracking others' mental states.
• Large Language Models (LLMs) are increasingly sophisticated, prompting comparisons with human cognitive functions.
• Debate exists on whether LLMs can exhibit human-like behavior in ToM tasks.

Purpose of the Study:

• To systematically compare human and LLM performance on a comprehensive suite of ToM tests.
• To investigate the capabilities of GPT and LLaMA2 families of LLMs in various ToM dimensions.
• To understand the nature of differences and similarities in ToM performance between humans and LLMs.

Main Methods:

• Administered a battery of ToM measurements to 1,907 human participants.
• Tested two LLM families (GPT and LLaMA2) repeatedly across the same ToM measures.
• Included tasks assessing false beliefs, indirect requests, irony, and faux pas detection.

Main Results:

• GPT-4 models performed at or above human levels on indirect requests, false beliefs, and misdirection.
• LLaMA2 outperformed humans only on faux pas detection, which was later found to be an illusory superiority.
• GPT-4's limitations stemmed from a conservative inference approach, not a fundamental lack of ability.

Conclusions:

• LLMs demonstrate behavior consistent with mentalistic inference, comparable to humans in certain ToM aspects.
• Systematic, nuanced testing is crucial for accurate comparisons between human and artificial intelligence.
• Differences in performance highlight the complexity of ToM and the need for further research into LLM cognition.