Evaluating the ability of large Language models to predict human social decisions

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Evolutionary Psychology

Background:

• Large language models (LLMs) demonstrate potential in predicting human decisions.
• Understanding AI's decision-making aligns with human psychology is crucial for AI development.

Purpose of the Study:

• To compare LLM (GPT-3.5, GPT-4, GPT-4o) predictions against human decisions in various scenarios.
• To investigate LLM's risk preference and sensitivity to social factors like kinship and group size.
• To identify criteria for a psychological Turing Test based on decision-making patterns.

Main Methods:

• Comparative analysis of 9,600 GPT-3.5/GPT-4 responses across 51 scenarios against 2,104 human participants' data.
• Evaluation of GPT-4o's 1,600 responses in eight social-group and kinship conditions.
• Application of an evolutionary-psychology framework and Prospect Theory.

Main Results:

• Humans exhibit greater sensitivity to kinship and group size in life-death decisions than LLMs.
• LLMs show different risk-seeking patterns compared to humans, often reversing Prospect Theory's value function.
• GPT-4 was consistently risk-averse, and GPT-4o made opposite predictions regarding group size effects compared to humans.

Conclusions:

• LLMs and humans exhibit distinct decision-making heuristics, particularly concerning social context and risk.
• Framing effects and social context-dependent risk preferences are key differentiators for a psychological Turing Test.
• Further research is needed to refine AI's alignment with human social cognition and decision-making.