Large language models can predict the results of social science experiments

Area of Science:

• Social and behavioral sciences
• Computational social science
• Artificial intelligence in research

Background:

• Growing interest in leveraging large language models (LLMs) for social and behavioral science.
• Limited understanding of LLMs' predictive capabilities for social science experimental outcomes, especially novel data.

Purpose of the Study:

• To assess the accuracy of LLMs in predicting outcomes of social science experiments.
• To evaluate LLM performance on data predating their training cutoffs and compare with human forecasts.
• To explore potential applications and risks of LLMs in social science research.

Main Methods:

• Compiled an archive of 70 preregistered US survey experiments (469 effects, 119,330 participants).
• Used GPT-4 and open-weight models to simulate participant responses to experimental stimuli.
• Inferred treatment effects from simulated responses and compared them with actual experimental results.

Main Results:

• GPT-4 predictions strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts.
• High correlations persisted for studies not publicly available before the LLM's training data cutoff.
• LLM predictions systematically overestimated effect sizes, though correlations were lower but comparable to experts in megastudies.

Conclusions:

• LLMs demonstrate significant potential to augment social science experimental methods and practices.
• Findings highlight the need for careful consideration of LLM applications, including potential biases and misuse.
• LLMs can aid in tasks like pilot testing, intervention selection, and identifying effects for replication.