Subjective recalibration of advisors' probability estimates

Area of Science:

• Decision analysis
• Behavioral economics
• Cognitive psychology

Background:

• Decision makers often rely on probabilistic forecasts from advisors.
• Understanding how decision makers interpret and use these forecasts is crucial for effective decision-making.
• The calibration of probability estimates (their statistical accuracy) is a key property that may influence decision makers.

Purpose of the Study:

• To investigate whether decision makers are sensitive to the calibration of probability estimates provided by advisors.
• To develop and test a model of how humans use probabilistic forecasts in decision-making.
• To compare the calibration of human decision makers against theoretical ideals.

Main Methods:

• Derivation of a Roughly Ideal Forecast Consumer (RIFC) model.
• Generalization of the RIFC model to incorporate human judgment limitations.
• An experiment where participants evaluated advisors based on their past probability estimates and outcomes.
• Analysis of participant confidence judgments in relation to advisor calibration.

Main Results:

• The generalized model accurately described participants' sensitivity to advisor calibration.
• Participants demonstrated appropriate sensitivity to the statistical properties of the advisors' forecasts.
• Individual models of participants were better calibrated than the participants themselves.
• However, even the best individual models were less calibrated than the theoretical RIFC.

Conclusions:

• Decision makers are indeed sensitive to the calibration of probability estimates.
• Human judgment in using forecasts is imperfect but can be modeled.
• Theoretical models provide a benchmark for ideal forecast utilization, highlighting areas for improvement in human decision-making.