A self-attention-based neural network for three-dimensional multivariate modeling and its skillful ENSO predictions

Large biases and uncertainties remain in real-time predictions of El Niño-Southern Oscillation (ENSO) using process-based dynamical models; recent advances in data-driven deep learning algorithms provide a promising mean to achieve superior skill in the tropical Pacific sea surface temperature (SST) modeling. Here, a specific self-attention-based neural network model is developed for ENSO predictions based on the much sought-after Transformer model, named 3D-Geoformer, which is used to predict three-dimensional (3D) upper-ocean temperature anomalies and wind stress anomalies. This purely data-driven and time-space attention-enhanced model achieves surprisingly high correlation skills for Niño 3.4 SST anomaly predictions made 18 months in advance and initiated beginning in boreal spring. Further, sensitivity experiments demonstrate that the 3D-Geoformer model can depict the evolution of upper-ocean temperature and the coupled ocean-atmosphere dynamics following the Bjerknes feedback mechanism during ENSO cycles. Such successful realizations of the self-attention-based model in ENSO predictions indicate its great potential for multidimensional spatiotemporal modeling in geoscience.