MSA-LR: Enhancing multi-scale temporal dynamics in multivariate time series forecasting with low-rank self-attention

Area of Science:

• Artificial Intelligence
• Machine Learning
• Time Series Analysis

Background:

• Deep learning models struggle to capture complex temporal dependencies across multiple scales in time series data.
• Existing architectures like LSTMs and Transformers have limitations in handling long-range dependencies and differentiating periodicities.

Purpose of the Study:

• To introduce MSA-LR (Multi-Scale Self-Attention with Low-Rank Approximation), a novel architecture for enhanced multivariate time series forecasting.
• To effectively capture multi-scale temporal dynamics and improve long-term forecasting accuracy.

Main Methods:

• Developed MSA-LR, a novel architecture utilizing a learnable scale weight matrix and low-rank approximations.
• Designed to directly model the influence of different temporal granularities (e.g., hourly, daily, weekly).
• Reduced computational complexity compared to standard self-attention for efficient processing of long time series.

Main Results:

• MSA-LR demonstrated competitive performance against state-of-the-art methods on diverse datasets (electricity load, traffic flow, air quality).
• Achieved notable improvements in long-term forecasting accuracy.
• Effectively discerned and leveraged periodic patterns at various resolutions.

Conclusions:

• MSA-LR successfully captures rich multi-scale temporal structures in real-world time series data.
• The model offers fine-grained control over multi-scale interactions and reduces computational cost.
• MSA-LR presents a promising advancement for accurate and efficient multivariate time series forecasting.