When are predictions useful? A new method for evaluating epidemic forecasts.

Area of Science:

• Epidemiology
• Public Health
• Biostatistics

Background:

• Future pandemics are inevitable, necessitating improved preparedness strategies.
• Evaluating the effectiveness of COVID-19 responses, particularly epidemiologic forecasting, is crucial.
• Assessing forecast accuracy is complex due to pandemic variability in space, time, and context.

Purpose of the Study:

• Introduce the Weighted Contextual Interval Score (WCIS) for retrospective interval forecast evaluation.
• Develop a method to incorporate contextual utility directly into forecast efficacy assessment.
• Provide a tool for analyzing prediction usefulness tailored to specific use cases.

Main Methods:

• The WCIS directly integrates contextual utility by defining a utility threshold parameter.
• This method extends the existing Weighted Interval Score (WIS) for probabilistic interval forecasts.
• The approach is generalized to accommodate the needs of epidemiological modeling.

Main Results:

• Applied WCIS to state-level and national hospitalization forecasts, demonstrating its utility.
• WCIS effectively captures both the relative quality and frequency of useful forecasts.
• Contextual normalization allows for comparison across variable pandemic scenarios and intuitive assessment of forecast quality.

Conclusions:

• WCIS offers a pragmatic, utility-based method for characterizing probabilistic predictions.
• It empowers non-expert practitioners and policymakers to analyze forecast insights.
• WCIS is designed for retrospective evaluation and not for competitive forecasting minimization.