Repetitive contrastive learning enhances Mamba's selectivity in time series prediction

Area of Science:

• Artificial Intelligence
• Machine Learning
• Time Series Analysis

Background:

• Long sequence prediction is a significant challenge in time series forecasting.
• Mamba-based models show promise but have limitations in selective focus and noise suppression.
• These limitations stem from the inherent selective abilities of Mamba architectures.

Purpose of the Study:

• To introduce Repetitive Contrastive Learning (RCL), a novel pretraining framework.
• To enhance the selective capabilities of Mamba models for improved temporal prediction.
• To boost the performance of Mamba-based models in long sequence forecasting tasks.

Main Methods:

• Developed a token-level contrastive pretraining framework (RCL).
• RCL pretrains a single Mamba block to strengthen selective abilities.
• Transferred pretrained parameters to initialize Mamba blocks in various backbone models.
• Employed sequence augmentation with Gaussian noise and contrastive learning strategies.

Main Results:

• RCL consistently improved the performance of backbone Mamba models.
• Achieved state-of-the-art results in long sequence prediction tasks.
• Demonstrated superior performance compared to existing methods.
• Proposed new metrics to quantify Mamba's selective capabilities.

Conclusions:

• Repetitive Contrastive Learning (RCL) effectively enhances Mamba's selective abilities.
• RCL significantly boosts temporal prediction performance in time series forecasting.
• The proposed framework offers a robust solution for long sequence prediction challenges.