Scoring epidemiological forecasts on transformed scales.

Area of Science:

• Epidemiology
• Computational Biology
• Biostatistics

Background:

• Epidemiological forecast evaluation is crucial for developing predictive models.
• Common scores like CRPS and WIS measure forecast distribution vs. observation.
• Direct application of scores to incidence counts can be problematic due to epidemic process characteristics.

Purpose of the Study:

• To investigate the benefits of transforming incidence counts before applying forecast evaluation scores.
• To demonstrate how transformations yield more meaningful and interpretable results.
• To highlight the advantages of using log-transformed data for forecast evaluation.

Main Methods:

• Applied Continuous Ranked Probability Score (CRPS) on log-transformed incidence counts.
• Utilized data and forecasts from the European COVID-19 Forecast Hub.
• Analyzed the impact of log(x+1) transformation on model rankings.

Main Results:

• Log-transformed CRPS offers a probabilistic relative error and reflects epidemic growth rate prediction accuracy.
• The logarithmic transformation yields expected CRPS values independent of the predicted quantity's magnitude.
• Transformations altered model rankings, emphasizing missed upward trends and downplaying missed peaks.

Conclusions:

• Appropriate data transformations, such as the natural logarithm, should be considered for evaluating infectious disease incidence forecast models.
• Logarithmic transformation provides a more robust and interpretable method for assessing model performance.
• This approach enhances the reliability of model comparisons and informs public health strategies.