Shaping Slow Manifolds in Equivariant Recurrent Neural Networks

Recordings of increasingly large neural populations have revealed that the firing of individual neurons is highly coordinated. When viewed in the space of all possible patterns, the collective activity forms nonlinear structures called neural manifolds. Because such structures are observed even at rest or during sleep, an important hypothesis is that activity manifolds may correspond to continuous attractors shaped by recurrent connectivity between neurons. Classical models of recurrent networks have shown that continuous attractors can be generated by specific symmetries in the connectivity. Although a variety of attractor network models have been studied, general principles linking network connectivity and the geometry of attractors remain to be formulated. Here, we address this question by using group representation theory to formalize the relationship between the symmetries in recurrent connectivity and the resulting fixed-point manifolds. We start by revisiting the classical ring model, a continuous attractor network generating a circular manifold. Interpreting its connectivity as a circular convolution, we draw a parallel with feedforward convolutional neural networks. Building on principles of geometric deep learning, we then generalize this architecture to a broad range of symmetries using group representation theory. Specifically, we introduce a new class of equivariant recurrent neural networks, where the connectivity is based on group convolution. Using the group Fourier transform, we reduce such networks to low-rank models. This formulation gives us a low-dimensional nonlinear description that can be fully analyzed to determine the symmetry, dimensionality, and stability of fixed-point manifolds. Our results underline the importance of stability considerations: for a connectivity with a given symmetry, depending on parameters, several manifolds with different topology can coexist, some stable and others consisting of saddle points. Our framework unifies a variety of existing models and offers a principled approach to build new types of continuous attractor models of neural systems.