Unsupervised learning of multiscale switching dynamical system models from multimodal neural data

Objective. Neural population activity often exhibits regime-dependent non-stationarity in the form of switching dynamics. Learning accurate switching dynamical system models can reveal how behavior is encoded in neural activity. Existing switching approaches have primarily focused on learning models from a single neural modality, either continuous Gaussian signals such as local field potentials (LFPs) or discrete Poisson signals such as spiking activity. However, multiple neural modalities are often recorded simultaneously to measure different spatiotemporal scales of brain activity, and all these modalities can encode behavior. Moreover, regime labels are typically unavailable in training data, posing a significant challenge for learning models of regime-dependent switching dynamics. These gaps highlight the need for a new unsupervised method that can learn switching dynamical system models for multiscale data and do so without requiring regime labels.Approach. We develop a novel unsupervised learning algorithm that learns the parameters of switching multiscale dynamical system (SMDS) models using only multiscale neural observations. Doing so, the algorithm can not only fuse multiscale neural information but also account for regime-dependent switches in multiscale neural dynamics.Main results. We demonstrate our method using both simulations and two distinct experimental datasets with multimodal spike-LFP observations during different motor tasks. We find that our SMDS models more accurately decode behavior than switching single-scale dynamical models, showing the success of multiscale neural fusion. Further, our models outperform stationary multiscale models, illustrating the importance of tracking regime-dependent non-stationarity in multimodal neural data.Significance. The developed unsupervised learning framework enables more accurate modeling of complex multiscale neural dynamics by leveraging information in multimodal recordings while incorporating regime switches. This approach holds promise for improving the performance and robustness of brain-computer interfaces over time and for advancing our understanding of the neural basis of behavior.