Studying and improving reasoning in humans and machines

Area of Science:

• Cognitive Psychology
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Cognitive psychology extensively studies human reasoning and decision-making, often focusing on bounded rationality.
• Large language models (LLMs) are increasingly sophisticated AI systems whose reasoning capabilities are of significant interest.
• Comparing AI and human cognition offers insights into both domains.

Purpose of the Study:

• To investigate and compare reasoning processes in large language models (LLMs) and humans.
• To utilize cognitive psychology tools to assess bounded rationality in both humans and LLMs.
• To identify similarities and differences in reasoning errors and performance between humans and LLMs.

Main Methods:

• Administered novel variants of classical cognitive experiments to human participants and various pretrained LLMs.
• Cross-compared the performance of human participants and LLMs on these cognitive tasks.
• Analyzed reasoning errors and response patterns to different prompting strategies.

Main Results:

• Most evaluated LLMs exhibited reasoning errors similar to heuristic-based human reasoning.
• Despite superficial similarities, significant differences were observed between human and LLM reasoning.
• More recent LLM releases demonstrated substantially reduced reasoning limitations compared to earlier models.
• Humans and LLMs showed differential responsiveness to specific prompting strategies designed to improve performance.

Conclusions:

• While LLMs can mimic certain human reasoning errors, their underlying cognitive processes may differ.
• Advancements in LLM technology are leading to more robust reasoning capabilities.
• Understanding the nuanced differences in human-AI interaction and prompting is crucial for both AI development and cognitive science.
• The study highlights the epistemological challenges and implications of comparing human and machine cognition.