Evaluating probabilistic forecasts with Bayesian signal detection models.

Area of Science:

• Decision Science
• Cognitive Psychology
• Forecasting Methodology

Background:

• Evaluating probabilistic forecasting systems and individual forecaster performance is crucial.
• Traditional methods struggle with subjective probabilities and sparse data in human forecasting.
• Signal detection theory (SDT) offers a framework to separate response bias from system accuracy.

Purpose of the Study:

• To adapt and apply signal detection theory (SDT) for evaluating probabilistic forecasts.
• To address challenges of using judged probabilities and sparse data within SDT.
• To provide a principled method for distinguishing forecaster response from system diagnosticity.

Main Methods:

• Developed a model of individual forecast generation from underlying representations.
• Employed Bayesian inference to estimate latent parameters of the forecasting process.
• Utilized estimated representations to derive SDT metrics like Receiver Operating Characteristic (ROC) curves and Area Under the Curve (AUC).

Main Results:

• Successfully applied SDT principles to probabilistic forecasts, overcoming limitations of binary decisions.
• Enabled the estimation of ROC curves and AUC for individual forecasters over time.
• Demonstrated the extension of the method to compare forecastability across different domains.

Conclusions:

• The proposed SDT-based approach provides a robust method for evaluating probabilistic forecasts.
• The method's reliance on ordinal properties of forecasts enhances its applicability.
• This framework can potentially improve decision-making processes informed by forecasts.