Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?

Area of Science:

• Artificial Intelligence
• Machine Learning
• Time Series Analysis

Background:

• Foundation models are increasingly used for time series data, achieving high predictive accuracy.
• Model calibration, assessing confidence in predictions, is vital for practical applications but underexplored in time series foundation models.

Purpose of the Study:

• To systematically evaluate the calibration properties of recent time series foundation models.
• To compare their calibration against traditional baseline models.
• To analyze calibration under varying prediction heads and long-term forecasting scenarios.

Main Methods:

• Evaluated five recent time series foundation models and two baseline models.
• Assessed calibration through systematic experiments, including confidence assessments (over-/under-confidence).
• Investigated the impact of different prediction heads and long-term autoregressive forecasting.

Main Results:

• Time series foundation models demonstrated consistently better calibration than baseline models.
• These models did not exhibit systematic over- or under-confidence.
• Performance contrasted with the frequent overconfidence observed in other deep learning models.

Conclusions:

• Time series foundation models offer superior calibration properties essential for trustworthy predictions.
• Their robust calibration makes them suitable for critical applications requiring reliable uncertainty quantification.
• Future research should continue exploring calibration in advanced time series modeling.