Geometry and efficiency of learned and reservoir recurrent dynamics in context-dependent integration-switching

Area of Science:

• Computational neuroscience
• Machine learning
• Dynamical systems

Background:

• Recurrent neural networks (RNNs) are crucial for sequential data processing.
• Two main RNN paradigms exist: fixed-reservoir computing and end-to-end trainable RNNs.
• Understanding their computational strategies encoded in state-space dynamics is key.

Purpose of the Study:

• To investigate fundamental differences between fixed-reservoir and trainable RNNs.
• To compare performance on a canonical context-dependent integration task.
• To uncover the mechanisms behind RNN efficiency.

Main Methods:

• Systematic comparison of echo-state networks, gated recurrent units, long short-term memory networks, and linear models.
• Analysis using dynamical systems and manifold theory.
• Intrinsic dimensionality and spectral analysis.

Main Results:

• Trainable RNNs achieve superior accuracy with significantly fewer parameters.
• Trainable networks sculpt internal dynamics into low-dimensional, task-aligned manifolds.
• Fixed reservoirs utilize high-dimensional, entangled representations, proving less efficient.

Conclusions:

• Learning to distill task structure into a compact representation is vital for efficient recurrent computation.
• Trainable RNNs demonstrate a more efficient computational strategy through dynamic sculpting.
• Findings link theoretical principles of universality to biological neural remapping.