Beyond forecast leaderboards: Measuring individual model importance based on contribution to ensemble accuracy

Area of Science:

• Forecasting science
• Ensemble modeling
• Statistical analysis

Background:

• Ensemble forecasts, combining multiple models, often yield superior predictions compared to individual models.
• Their application is expanding in decision-making and policy planning across diverse fields.
• Understanding individual model contributions within ensembles is crucial for optimizing collaborative forecasting efforts.

Purpose of the Study:

• To propose practical methods for quantifying the value of individual models within an ensemble.
• To analyze the relationship between these value metrics, forecast accuracy, and error similarity.
• To provide insights into unique model contributions not captured by standard accuracy metrics.

Main Methods:

• Developed a leave-one-model-out algorithm to assess ensemble performance changes when a model is removed.
• Implemented a leave-all-subsets-of-models-out algorithm based on Shapley value for comprehensive model contribution assessment.
• Conducted analytical explorations and simulations to examine metric relationships with forecast accuracy and error patterns.

Main Results:

• The proposed methods effectively measure the marginal contribution of each model to ensemble performance.
• Demonstrated that these metrics offer insights beyond standard accuracy, revealing unique model characteristics.
• Illustrated the application using US COVID-19 death probabilistic forecasts, highlighting model value in a real-world scenario.

Conclusions:

• The developed methods provide a robust framework for evaluating individual model importance in ensemble forecasting.
• These techniques enhance the understanding of collaborative forecasting dynamics and model selection.
• Offers valuable insights for improving ensemble construction and leveraging diverse modeling strengths.