Validation framework for epidemiological models with application to COVID-19 models.

Area of Science:

• Epidemiology
• Mathematical Modeling
• Public Health

Background:

• Mathematical models were crucial during the COVID-19 pandemic for resource demand prediction.
• Limited information exists on the reliability and predictive accuracy of these COVID-19 models.

Purpose of the Study:

• To present a general model validation framework for epidemiological models.
• To focus on predictive capability for decision-making end-users.
• To systematically account for multiple releases and localities of COVID-19 models.

Main Methods:

• Developed a framework using validation scores to quantify model accuracy for peak date, peak magnitude, recovery rate, and monthly cumulative counts.
• Applied the framework to retrospectively assess death and hospitalization predictions from COVID-19 models.
• Ensured sufficient publicly available data for model assessment.

Main Results:

• The most accurate models predicted peak death dates with errors under 15 days, 3-6 weeks in advance.
• Death peak magnitude errors were around 50% when predicted 3-6 weeks prior.
• Hospitalization predictions were less accurate than death predictions, with high regional variability.

Conclusions:

• The validation framework provides valuable insights into the predictive accuracy of epidemiological models.
• It can be used to evaluate new models and support existing methodologies in future epidemics.
• This aids in informed, model-based public health decision-making.