What is a Good Calibration Question?

Area of Science:

• Decision Analysis
• Cognitive Science
• Information Science

Background:

• Performance-weighted aggregation of expert judgments can improve upon equal weights.
• Obtaining validated, relevant calibration questions is often challenging.
• The utility of using less relevant calibration questions for weighting is unclear.

Purpose of the Study:

• To evaluate the out-of-sample performance of weighted aggregations versus equal weights.
• To investigate the impact of including irrelevant calibration questions on weighted aggregation performance.
• To assess expert adaptability across different domains.

Main Methods:

• Comparison of weighted aggregation (Classical Model) against equal weights.
• Inclusion of a significant number of "irrelevant" calibration questions.
• Analysis of performance on combined scores and statistical accuracy (calibration).

Main Results:

• Performance-weighted aggregations outperformed equal weights on the combined score, but not on statistical accuracy.
• No significant performance difference was observed when weights were developed on relevant versus irrelevant questions.
• In-sample validation did not reliably predict out-of-sample performance on irrelevant questions.

Conclusions:

• If relevant calibration questions are unavailable, using equal weights and alternative judgment improvement techniques is advisable.
• Performance-weighted aggregation has limitations in predictive accuracy and adaptability of expertise across domains.
• Further research is needed on the effect of question type on the reliability of performance-weighted aggregations.