Basic Baseline model design choices can substantially influence performance in collaborative forecast hubs

Area of Science:

• Epidemiology
• Public Health
• Computational Biology

Background:

• Outbreak forecasting aids public health decision-making during epidemics.
• Collaborative forecast hubs are the standard for real-time epidemic predictions.
• Baseline models serve as crucial benchmarks within these hubs but their design is often subjective and understudied.

Purpose of the Study:

• To evaluate the impact of subjective Baseline model design decisions on forecast performance.
• To identify optimal Baseline model specifications for trend forecasting.
• To compare trend baseline models with seasonal baseline models for specific surveillance targets.

Main Methods:

• Retrospective forecasting was performed using multiple years of data for COVID-19, influenza, and RSV hospital admissions, and weighted influenza-like illness (wILI).
• Three key Baseline specification decisions were evaluated: amount of historical data, data transformation, and forecast variant (flatline vs. drift).
• Trend baseline models were compared to a seasonal baseline model for wILI data.

Main Results:

• Model specification significantly altered forecast performance across all targets.
• An optimal performing model was identified as a flatline model using 6-12 transformed recent observations.
• This optimal model outperformed the current standard Baseline by an average of 9.6% and a seasonal baseline by 32.3%.

Conclusions:

• Subjective Baseline design choices materially influence forecast accuracy and model rankings within collaborative hubs.
• Increased transparency in Baseline model specifications is necessary.
• Routine inclusion of multiple benchmark models in forecast hubs is recommended.