Quantifying machine influence over human forecasters.

Area of Science:

• Geopolitical forecasting
• Cognitive science
• Artificial intelligence

Background:

• Crowdsourcing and machine learning (ML) offer distinct advantages for predicting geopolitical outcomes.
• Hybrid approaches combining human forecasters with ML models show promise for enhanced accuracy.
• Human mistrust and underutilization of ML models in forecasting are well-documented challenges.

Purpose of the Study:

• To analyze the effectiveness of hybrid human-forecaster and ML model systems.
• To develop a model for estimating human trust in machine predictions.
• To investigate how forecasters' prior beliefs and cognitive biases influence their use of ML models.

Main Methods:

• Utilized prior human forecasts (without ML models) to establish baseline beliefs.
• Developed a model to quantify the trust (weights) humans assign to ML models.
• Analyzed forecaster decision-making processes, including model selection and trust assignment.

Main Results:

• Forecasters exhibit low trust in ML models, treating them akin to expert advisors.
• Trust in ML models is higher among top-performing forecasters and when models are expected to perform well.
• Forecasters tend to favor ML models aligning with their prior beliefs over objective model outputs.

Conclusions:

• Hybrid systems can enhance human judgment in forecasting, but require careful consideration of trust dynamics.
• Cognitive biases, such as confirmation bias, significantly impact how humans integrate ML model outputs.
• Future research should focus on mitigating biases and fostering appropriate trust in human-ML collaborative forecasting.