A framework for evaluating epidemic forecasts.

Area of Science:

• Computational epidemiology
• Mathematical modeling
• Public health surveillance

Background:

• Numerous epidemic forecasting methods exist, often categorized as deterministic vs. probabilistic or comparative vs. generative.
• Comparative methods assess outbreak data against model outputs for trend prediction.
• A key challenge is the absence of standardized evaluation metrics for algorithm and configuration selection.

Purpose of the Study:

• To introduce a comprehensive evaluation framework for epidemic forecasting models.
• To define epidemic features (Epi-features) and associated error measures for forecast assessment.
• To address the need for nuanced evaluation beyond short-term predictions.

Main Methods:

• Developed an evaluation framework integrating diverse features, error measures, and ranking schemas.
• Defined and utilized specific epidemic features (Epi-features) to characterize forecast outputs.
• Applied the framework to evaluate six forecasting methods for influenza prediction in the United States, focusing on long-term predictions.

Main Results:

• Different error measures yield varying rankings for the same Epi-feature.
• No single forecasting method consistently outperformed others across all Epi-features and error measures.
• Consensus Ranking schemas were proposed to synthesize individual rankings and account for multiple error measures.

Conclusions:

• A nuanced approach is necessary for evaluating epidemic forecasts.
• Multiple forecasting methods should be combined for comprehensive predictions.
• The proposed evaluation framework offers significant value to the computational epidemiology community.