Language models, like humans, show content effects on reasoning tasks

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Linguistics

Background:

• Abstract reasoning is crucial for intelligent systems, yet both humans and large language models (LMs) exhibit imperfections.
• Human reasoning is influenced by real-world knowledge and beliefs, leading to 'content effects' where semantic content aids logical inference.
• Understanding these content effects in LMs is key to understanding their cognitive parallels with humans.

Purpose of the Study:

• To investigate whether large language models (LMs) exhibit content effects in their reasoning, similar to humans.
• To compare LM and human performance across diverse logical reasoning tasks.
• To explore the implications of these findings for human cognition and LM development.

Main Methods:

• Evaluated state-of-the-art LMs and human participants on three reasoning tasks: natural language inference, syllogistic validity judgment, and the Wason selection task.
• Analyzed accuracy patterns and lower-level features, such as the relationship between LM confidence and human response times.
• Compared qualitative patterns of reasoning between LMs and humans across tasks.

Main Results:

• LMs, like humans, demonstrate improved accuracy when the semantic content of a reasoning task supports logical inferences.
• Parallels were observed in accuracy and in the relationship between LM confidence and human response times.
• Significant differences emerged on the Wason selection task, where humans performed substantially worse than LMs and showed distinct error patterns.

Conclusions:

• Large language models (LMs) mirror human 'content effects' in logical reasoning, suggesting shared mechanisms or influences.
• The findings provide insights into the cognitive underpinnings of human content effects and the factors shaping LM reasoning.
• Differences in performance, particularly on the Wason task, highlight areas where LM and human reasoning diverge, warranting further investigation.