Crowdsourced versus large language models forecasting: evidence for the accuracy-correlation effect.

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computational Social Science

Background:

• Crowdsourced forecasting has historically surpassed individual predictions.
• Large language models (LLMs) represent a novel form of collective intelligence (CI) by aggregating human knowledge.
• The relationship between LLM predictive accuracy and human-AI correlation is a key area of investigation.

Purpose of the Study:

• To investigate the accuracy-correlation effect (ACE) in large language models (LLMs).
• To determine if the correlation between LLM accuracy and human predictions exceeds what is expected from tracking the same underlying truth.
• To assess the implications of ACE on the value of human input in hybrid forecasting ensembles.

Main Methods:

• Utilized 76 model × prompt forecast sets from 16 LLMs on 580 ForecastBench questions.
• Computed LLM accuracy and correlations with human aggregates (superforecasters, general public).
• Employed linear mixed-effects models to analyze the association between LLM accuracy and human-AI correlation across different question types (databases, prediction markets).

Main Results:

• A robust positive association was found between LLM accuracy and human-AI correlation, significantly exceeding independent-errors predictions.
• Correlations were lower with superforecasters compared to the general public.
• Correlations were weaker for prediction market questions than for database questions.

Conclusions:

• The findings support the accuracy-correlation effect (ACE), indicating that increasing LLM accuracy leads to higher correlation with human predictions.
• The observed correlation increase reflects more than just improved signal tracking, suggesting a complex interaction.
• While ACE may reduce optimal human weights in data-rich settings, human judgment remains crucial for contextual reasoning.