Mathematically aggregating experts' predictions of possible futures

Area of Science:

• Decision Science
• Cognitive Science
• Social Psychology

Background:

• Structured protocols facilitate transparent aggregation of probabilistic predictions from multiple experts.
• Mathematical aggregation rules offer objective methods for deriving group predictions, evaluated by accuracy, calibration, and informativeness.
• Performance-based weighting is ideal when expert performance is measurable, but alternative methods are needed otherwise.

Purpose of the Study:

• To develop and compare novel aggregation methods for expert judgments.
• To investigate leveraging individual knowledge and behavioral measures for enhanced group predictions.
• To identify aggregation strategies that yield superior accuracy, calibration, and informativeness.

Main Methods:

• Developed a suite of aggregation methods differentially weighting expert estimates based on reasoning, engagement, openness, informativeness, prior knowledge, and estimate characteristics.
• Evaluated the performance of these methods using three distinct datasets.
• Compared novel methods against established benchmarks like simple average and median.

Main Results:

• Most aggregation methods showed similar accuracy, calibration, and informativeness.
• A few novel methods consistently outperformed others, identifying them as superior or inferior.
• The majority of developed methods surpassed simple average and median benchmarks.

Conclusions:

• Weighting expert estimates by knowledge and behavioral proxies can improve group judgment performance.
• Novel aggregation strategies offer advantages over traditional methods for combining expert predictions.
• Further research can refine these methods for optimal decision-making support.