Bridging Distribution Gaps in Time Series Foundation Model Pretraining With Prototype-Guided Normalization.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Foundation models (FMs) excel in machine learning due to large-scale pretraining.
• Pretraining on diverse datasets, especially time series (TS) data, faces challenges from data distribution mismatches.
• Existing methods struggle to adapt pretrained models to the heterogeneity of TS data characteristics.

Purpose of the Study:

• To propose a novel domain-aware adaptive normalization strategy for transformer-based foundation models.
• To address the significant challenges posed by data distribution shifts in time series pretraining.
• To enhance the alignment of pretrained representations with downstream tasks for improved performance.

Main Methods:

• Introduced a prototype-guided dynamic normalization mechanism (ProtoNorm) to replace traditional LayerNorm.
• Learned prototypes to represent distinct data distributions within TS data.
• Utilized sample-to-prototype affinity to dynamically select appropriate normalization layers.

Main Results:

• The proposed ProtoNorm method significantly outperforms conventional pretraining techniques on diverse downstream TS tasks.
• Effectively mitigated the adverse effects of distribution shifts encountered during pretraining.
• Demonstrated robustness and generalizability across various real-world TS benchmarks.

Conclusions:

• ProtoNorm offers an effective solution for handling data distribution heterogeneity in TS foundation models.
• The method seamlessly integrates into existing transformer architectures with a single line of code.
• Advances the development of more versatile and robust time series foundation models.