Combining experts: decomposition and aggregation order.

Area of Science:

• Decision Analysis
• Expert Elicitation
• Quantitative Methods

Background:

• Expert elicitation is crucial for decision-making under uncertainty.
• Decomposition of quantities is a common technique to simplify complex elicitation tasks.
• The optimal aggregation strategy for expert judgments after decomposition is not well-established.

Purpose of the Study:

• To investigate the impact of the timing of expert aggregation on the accuracy of elicited quantities.
• To compare two distinct expert judgment aggregation strategies: combining experts before recomposition versus after recomposition.

Main Methods:

• The study employed both simulation modeling and application to real-world data.
• Expert judgments on decomposed quantities were aggregated at different stages of the recomposition process.
• Error metrics and variance were used to evaluate the performance of each aggregation strategy.

Main Results:

• Combining experts earlier in the process, before recomposing quantities, consistently resulted in lower errors.
• This early aggregation strategy also led to a reduction in the variance of the elicited quantities.
• While simulation indicated potentially small average differences, real-world data demonstrated significant impacts on individual decision problems.

Conclusions:

• Aggregating expert judgments prior to recomposing decomposed quantities is a more effective strategy for improving accuracy and reducing uncertainty.
• The findings have practical implications for designing more robust expert elicitation protocols.
• Decision-makers should consider the timing of expert aggregation to optimize the reliability of elicited information.