Reducing overconfidence in the interval judgments of experts

Area of Science:

• Decision Analysis
• Risk Assessment
• Expert Elicitation

Background:

• Expert opinion is crucial for risk analysis with limited data.
• Subjective confidence intervals from experts often exhibit significant overconfidence.
• Overconfidence can lead to inaccurate risk assessments.

Purpose of the Study:

• To investigate the impact of elicitation question format on expert overconfidence.
• To compare a 3-point elicitation procedure with a novel 4-step procedure.
• To determine if the 4-step procedure reduces overconfidence in interval predictions.

Main Methods:

• Three studies were conducted with experts in epidemiology, public health, ecology, and biology.
• Experts provided interval predictions for disease rates and population sizes.
• A meta-analysis combined results, comparing a 3-point (lower, upper, best guess) with a 4-step (adding confidence rating) procedure.

Main Results:

• A meta-analysis revealed an average overconfidence of 11.9% (68.1% hit rate for 80% intervals), a reduction from previous studies.
• Studies 2 and 3 indicated the 4-step procedure was more effective at reducing overconfidence than the 3-point procedure (Cohen's d = 0.61).

Conclusions:

• The 4-step expert elicitation procedure shows promise in mitigating overconfidence.
• This method can improve the reliability of expert-based risk assessments in various scientific fields.
• Refining elicitation techniques is vital for more accurate data-driven decision-making.