T-SHRED: symbolic regression for regularization and model discovery with transformer shallow recurrent decoders

Shallow recurrent decoders (SHRED) are effective for system identification and forecasting from sparse sensor measurements. Such models are lightweight and computationally efficient, allowing them to be trained on consumer laptops. SHRED-based models rely on recurrent neural networks (RNNs) and a simple multi-layer perceptron (MLP) for temporal encoding and spatial decoding, respectively. Despite the relatively simple structure of SHRED, they are able to predict chaotic dynamical systems on different physical, spatial and temporal scales directly from a sparse set of sensor measurements. In this work, we modify SHRED by leveraging transformer-SHRED (T-SHRED) embedded with symbolic regression for the temporal encoding, circumventing auto-regressive long-term forecasting for physical data. This is achieved through a new sparse identification of nonlinear dynamics (SINDy) attention mechanism into T-SHRED to impose sparsity regularization on the latent space, which also allows for immediate symbolic interpretation. Symbolic regression improves model interpretability by learning and regularizing the dynamics of the latent space during training. We analyse the performance of T-SHRED on three different dynamical systems ranging from low-data to high-data regimes. This article is part of the discussion meeting issue 'Symbolic regression in the physical sciences'.