Model-based ensembles: Lessons learned from retrospective analysis of COVID-19 infection forecasts across 10...

Area of Science:

• Epidemiology
• Mathematical Modeling
• Data Science

Background:

• Mathematical models are crucial for understanding COVID-19 progression and intervention effects.
• Forecasts from these models, regardless of complexity, are subject to substantial uncertainty.
• Data quality, including testing and reporting accuracy, significantly impacts model reliability.

Purpose of the Study:

• To analyze the aggregated effect of systematic biases on ensemble-based COVID-19 model forecasts.
• To compare the accuracy of susceptible-infected-removed (SIR) and Holt-Winters time series models.
• To assess the sensitivity of model parameters and forecasting skill across different countries.

Main Methods:

• Comparative and retrospective analyses of active COVID-19 infections.
• Re-forecasting using susceptible-infected-removed (SIR) and Holt-Winters models.
• Systematic estimation of model parameters from varying data subsets and time windows.

Main Results:

• Considerable variations in forecasting skill were observed among ten highly affected countries.
• Individual model predictions demonstrated high sensitivity to parameter assumptions.
• Significant forecasting skill was primarily confirmed for short-term predictions (up to a few weeks).

Conclusions:

• COVID-19 model forecasts are highly sensitive to data quality and parameter choices.
• Ensemble-based forecasting skill varies significantly by country and time period.
• Accurate short-term forecasting is achievable, but long-term predictions remain challenging.