Single-unit activations confer inductive biases for emergent circuit solutions to cognitive tasks

Area of Science:

• Computational neuroscience
• Artificial intelligence
• Machine learning

Background:

• Recurrent neural networks (RNNs) are widely used to model brain dynamics and population-level computations.
• It is commonly assumed that the choice of nonlinear activation functions in RNN units does not influence emergent task solutions.

Purpose of the Study:

• To investigate how single-unit activation functions in RNNs influence neural representations, dynamics, and task solutions.
• To determine if different activation functions lead to qualitatively distinct circuit solutions and generalization behaviors.

Main Methods:

• Utilized trained recurrent neural networks (RNNs) with varying nonlinear activation functions.
• Employed a model distillation approach to analyze differences in neural representations and dynamics.
• Evaluated generalization performance on out-of-distribution inputs.

Main Results:

• Single-unit activation functions impose inductive biases that shape neural population trajectories, single-unit selectivity, and fixed-point configurations.
• Distinct activation functions result in qualitatively different circuit solutions for cognitive tasks.
• These differences lead to varied generalization behaviors on unseen data.

Conclusions:

• Activation function choice is not a minor architectural detail but a significant factor conferring inductive biases.
• Different RNN architectures yield distinct solutions to cognitive tasks, impacting generalization.
• The study raises questions about which RNN architectures best model biological neural mechanisms for task execution.