MARINE-Transformer: A General-purpose framework for multivariate ocean time series analysis

Despite the success of self-supervised pre-training in texts and images, its application to multivariate time series from oceanographic IoT devices lags behind tailored methods for critical tasks like forecasting, data imputation, and anomaly detection. To address this, we propose a general-purpose framework, named MARINE-Transformer. It conducts task-agnostic pre-training on large-scale unlabeled data to learn general representations, which are then adapted to specific downstream tasks through a parameter-efficient fine-tuning process. We identify two fundamentally heterogeneous dependencies in ocean data that demand separate modeling strategies: universal temporal dynamics, which are shared across different variables, and specific cross-variable physics, which govern their interactions. Our novel univariate-to-multivariate paradigm is devised to tackle this dichotomy, addressing each type of dependency in a dedicated stage. In the univariate pre-training stage, a Masked AutoEncoder (MAE) learns the intrinsic temporal dynamics of individual ocean variables by deliberately discarding their interdependencies. Subsequently, for multivariate fine-tuning, downstream tasks are formulated as specific mask-reconstruction problems, and the pre-trained encoder is leveraged to construct a dependency graph that explicitly models the complex interactions between variables. Experiments on multiple real-world oceanographic datasets show our framework achieves state-of-the-art performance in forecasting, imputation and anomaly detection.