Evaluation of short-term multi-target respiratory forecasts over winter 2024-25 in England using sub-ensemble...

Area of Science:

• Epidemiology
• Computational Biology
• Public Health

Background:

• Assessing epidemic forecasting models and ensembles is crucial for public health preparedness.
• Quantifying the impact of individual models on ensemble performance across diverse targets and scales remains a challenge.

Purpose of the Study:

• To evaluate the performance of operational ensemble forecasts for Influenza and COVID-19 hospital admissions in England.
• To quantify the contribution of individual models to ensemble performance using retrospective simulations and Pareto analysis.

Main Methods:

• Utilized Winter 2024-25 forecasts for Influenza and COVID-19 hospital admissions.
• Employed per capita weighted interval score (pcWIS) for counts and ranked probability score (RPS) for trend direction.
• Applied generalized additive models (GAMs) and Pareto analysis to assess sub-ensemble performance and model contributions.

Main Results:

• Operational ensembles showed mixed results: improved Influenza pcWIS but worse RPS compared to sub-ensembles.
• COVID-19 operational ensembles underperformed sub-ensembles significantly on both pcWIS and RPS.
• Despite underperformance against sub-ensembles, operational ensembles were generally better than individual models for both diseases.

Conclusions:

• Operational ensemble models for Influenza and COVID-19 did not consistently outperform simpler sub-ensembles, indicating potential issues with ensemble construction or model selection.
• Pareto analysis revealed trade-offs between optimizing for forecast accuracy (count scores) and directional accuracy (trend scores).
• Further research is needed to optimize ensemble methodologies for improved epidemic forecasting across various diseases and scales.