(Ir)rationality and cognitive biases in large language models.

Area of Science:

• Cognitive Psychology
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) are known to inherit human biases from training data.
• The extent to which LLMs exhibit rational reasoning, and how it compares to human rationality, is not well understood.

Purpose of the Study:

• To investigate whether large language models (LLMs) display rational reasoning.
• To compare the reasoning patterns of LLMs with human cognitive biases using established psychological tasks.

Main Methods:

• Evaluated seven different large language models (LLMs).
• Utilized tasks commonly found in cognitive psychology literature to assess reasoning.
• Analyzed the nature of errors and inconsistencies in LLM responses.

Main Results:

• LLMs demonstrate irrationality in reasoning tasks, mirroring human performance in some aspects.
• LLM irrationality manifests differently from human cognitive biases, with errors often unique to the models.
• LLMs exhibit significant response inconsistency, adding another layer of irrationality.

Conclusions:

• LLMs do not replicate human-like irrationality in reasoning tasks.
• LLM reasoning is characterized by unique error patterns and notable response inconsistency.
• The study proposes a methodological framework for assessing and comparing LLM capabilities, specifically rational reasoning.