Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer.

Area of Science:

• Machine Learning
• Time Series Analysis
• Signal Processing

Background:

• Self-supervised representation learning is crucial for time series data due to limited labeled samples.
• Existing contrastive learning methods face challenges with sampling policies and pair construction.
• Effective modeling of temporal-spectral correlations is underexplored for improving time series representations.

Purpose of the Study:

• To propose a novel self-supervised learning framework for time series representation.
• To address limitations of existing methods regarding sampling and temporal-spectral correlation modeling.
• To enhance the capacity and discriminative power of learned time series representations.

Main Methods:

• Introduced the Cross Reconstruction Transformer (CRT) utilizing a cross-domain dropping-reconstruction task.
• Employed Fast Fourier Transform (FFT) to obtain the frequency domain and randomly dropped patches in both time and frequency domains.
• Utilized a Transformer architecture for Dropped Temporal-Spectral Modeling, incorporating Instance Discrimination Constraint (IDC) and Curriculum Learning (CL).

Main Results:

• CRT achieved state-of-the-art performance, outperforming existing methods by 2%-9% across multiple real-world datasets.
• The Dropped Temporal-Spectral Modeling effectively captures cross-domain correlations.
• Instance Discrimination Constraint (IDC) sharpened decision boundaries, and Curriculum Learning (CL) improved pretraining robustness.

Conclusions:

• The proposed Cross Reconstruction Transformer (CRT) offers a superior approach to self-supervised time series representation learning.
• CRT effectively models temporal-spectral correlations and overcomes limitations of previous methods.
• The method demonstrates significant performance gains and robustness in real-world applications.