Case study in evaluating time series prediction models using the relative mean absolute error.

Area of Science:

• Statistics
• Epidemiology
• Computational Science

Background:

• Statistical prediction models are crucial for real-world decision-making.
• Rigorous validation of these models is essential before practical application to ensure accuracy.

Purpose of the Study:

• To present a computationally simple and intuitively interpretable framework for evaluating time series predictions.
• To introduce the relative mean absolute error metric as a tool for performance benchmarking.

Main Methods:

• Developed a framework centered on the relative mean absolute error (MAE) metric.
• Applied the framework to compare candidate model predictions against naïve reference models for single time series.
• Extended the framework for comparing multiple models across different datasets in multi-time series applications.

Main Results:

• The relative MAE metric offers standardized performance benchmarks by comparing predictions to naïve models.
• The framework facilitates cross-dataset comparisons of predictive performance for multiple models.
• Demonstrated utility and interpretability of relative MAE in a dengue hemorrhagic fever incidence prediction case study.

Conclusions:

• The relative MAE metric provides a practical and interpretable method for evaluating time series prediction models.
• Relative performance metrics offer practical advantages over absolute metrics when assessing prediction accuracy.
• The proposed framework enhances the reliability of statistical predictions in real-world settings.