Challenges of COVID-19 Case Forecasting in the US, 2020-2021

Area of Science:

• Epidemiology
• Public Health
• Data Science

Background:

• Accurate forecasting of COVID-19 trends was crucial for pandemic response and planning.
• The US COVID-19 Forecast Hub coordinated numerous forecasting efforts.
• Evaluating forecast performance across different epidemic phases is essential.

Purpose of the Study:

• To assess the accuracy and reliability of COVID-19 case forecasts submitted to the US COVID-19 Forecast Hub.
• To compare the performance of different forecasting models and approaches.
• To identify factors influencing forecast accuracy, such as epidemic phase and jurisdiction size.

Main Methods:

• Analysis of approximately 9.7 million weekly state-level COVID-19 case forecasts (1-4 weeks ahead) from 24 teams (August 2020 - December 2021).
• Assessment of prediction interval coverage and weighted interval scores (WIS), adjusting for missing data.
• Use of Gaussian generalized estimating equation (GEE) models to evaluate skill across epidemic phases defined by the effective reproduction number.

Main Results:

• Significant variation in forecast skill across individual models; ensemble-based forecasts generally outperformed others.
• Forecast skill was higher for larger jurisdictions (states vs. counties).
• Forecasts performed poorly during periods of rapid case increases or decreases, with 95% prediction interval coverage falling below 50% during major waves (Winter 2020, Delta, Omicron).

Conclusions:

• While most COVID-19 case forecasts surpassed baseline models, even the best were unreliable during critical epidemic phases.
• Limitations in current case forecasting necessitate cautious interpretation and the use of diverse indicators for decision-making.
• Future research should focus on integrating real-time data, improving phase-specific performance, quantifying forecast confidence, and ensuring spatial coherence.