PRformer: Pyramidal recurrent transformer for multivariate time series forecasting.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Time Series Analysis

Background:

• Transformer models, while powerful, lack inherent temporal order awareness due to their self-attention mechanism.
• Reliance on positional embeddings limits Transformer effectiveness in time series prediction, especially with extended lookback windows.
• Existing methods struggle to capture complex temporal dependencies crucial for accurate forecasting.

Purpose of the Study:

• To develop an improved method for encoding temporal order in Transformer models for time series prediction.
• To enhance the representation of temporal sequences, particularly for longer lookback windows.
• To integrate a novel embedding technique with Transformer architecture for superior predictive performance.

Main Methods:

• Introduced Pyramid RNN embeddings (PRE) combining pyramidal 1D convolutional layers and Recurrent Neural Networks (RNNs).
• PRE constructs multiscale convolutional features preserving temporal order and learns sequence-order-sensitive representations.
• Integrated PRE with a standard Transformer encoder to create the PRformer model for multivariate time series prediction.

Main Results:

• The PRformer model demonstrated significant performance enhancements compared to standard Transformer approaches.
• Achieved state-of-the-art results on various real-world time series datasets.
• Effectively leveraged longer lookback windows, showcasing the robustness of the proposed temporal representations.

Conclusions:

• The proposed Pyramid RNN embeddings (PRE) effectively address the temporal order limitations of standard Transformer models.
• Integrating PRE significantly boosts Transformer performance in time series prediction tasks, especially with long lookback data.
• Robust temporal representations are critical for maximizing the potential of Transformer architectures in predictive modeling.