Bridging Distribution Gaps in Time Series Foundation Model Pretraining With Prototype-Guided Normalization

Foundation models (FMs) have achieved remarkable success across diverse machine learning domains through large-scale pretraining on large, diverse datasets. However, pretraining on such datasets introduces significant challenges due to substantial mismatches in data distributions, a problem particularly pronounced with time series (TS) data. In this article, we tackle this issue by proposing a domain-aware adaptive normalization strategy within the transformer architecture. Specifically, we replace the traditional LayerNorm with a prototype-guided dynamic normalization mechanism (ProtoNorm), where learned prototypes encapsulate distinct data distributions, and sample-to-prototype affinity determines the appropriate normalization layer. This mechanism effectively captures the heterogeneity of TS characteristics, aligning pretrained representations with downstream tasks. Through comprehensive empirical evaluation, we demonstrate that our method significantly outperforms conventional pretraining techniques across diverse downstream tasks, while effectively mitigating the adverse effects of distribution shifts during pretraining. Incorporating ProtoNorm is as simple as replacing a single line of code. Extensive experiments on diverse real-world TS benchmarks validate the robustness and generalizability of our approach, advancing the development of more versatile TS FMs.