RoGAtten: Rotary gated linear attention for multivariate time series forecasting.

Area of Science:

• Computer Science
• Artificial Intelligence
• Network Engineering

Background:

• Internet of Things (IoT) generates vast long-term time series data.
• Accurate network traffic prediction is crucial for security and management.
• Transformer models offer high prediction accuracy but struggle with attention mechanism efficiency.

Purpose of the Study:

• To enhance attention mechanisms for improved network traffic forecasting.
• To develop a novel model that balances expressivity and computational efficiency.
• To leverage insights from state space models like Mamba for attention enhancement.

Main Methods:

• Theoretically proved linear attention with rotary positional embeddings can resemble Mamba.
• Designed a scalable rotary position embedding (SRoPE) mechanism with a scaling factor.
• Proposed Rotary Gated linear Attention (RoGAtten) for multivariate time series forecasting.
• Integrated SRoPE to provide series-wise identifiers and adjust inter-variable interactions.

Main Results:

• SRoPE confers forget-gate-like capability, enhancing expressiveness over previous attention variants.
• RoGAtten effectively captures inter-series dependencies in multivariate time series.
• Experiments on 8 real-world datasets show significant performance improvements.
• RoGAtten reduced Mean Squared Error (MSE) by 3.85% and Mean Absolute Error (MAE) by 1.71% compared to state-of-the-art methods.

Conclusions:

• RoGAtten offers a superior approach to multivariate time series forecasting in network traffic prediction.
• The proposed SRoPE mechanism enhances model adaptability and alignment with domain knowledge.
• This research provides a computationally efficient and highly expressive alternative to existing attention mechanisms for time series analysis.