Forecast evaluation for data scientists: common pitfalls and best practices.

Area of Science:

• Data Science
• Machine Learning
• Statistics

Background:

• Machine Learning (ML) and Deep Learning (DL) show increasing competitiveness in time series forecasting with large datasets.
• Non-stationarities in time series data pose significant challenges for data-driven ML models.
• Forecast evaluation methodologies are not widely understood within the ML community, leading to potential misinterpretations of model performance.

Purpose of the Study:

• To provide a tutorial-like compilation of forecast evaluation details tailored for ML researchers.
• To bridge the knowledge gap between traditional forecasting methods and state-of-the-art ML techniques.
• To address flawed evaluation practices in ML that can lead to spurious conclusions about model competitiveness.

Main Methods:

• Elaboration on problematic time series characteristics like non-normality and non-stationarities.
• Outline of best practices in forecast evaluation, including data partitioning, error calculation, and statistical testing.
• Guidelines for selecting appropriate error measures based on dataset characteristics.

Main Results:

• Identification of common pitfalls in forecast evaluation stemming from time series properties.
• Demonstration of how flawed evaluation practices can lead to misleading conclusions about ML model performance.
• A structured approach to forecast evaluation for ML practitioners.

Conclusions:

• Accurate forecast evaluation is crucial for the reliable application of ML and DL in time series forecasting.
• Understanding time series characteristics and adopting rigorous evaluation practices are essential for ML researchers.
• This work aims to improve the robustness and interpretability of ML-based forecasting models.