How connectivity structure shapes rich and lazy learning in neural circuits.

Area of Science:

• Theoretical neuroscience
• Deep learning
• Computational neuroscience

Background:

• Neural network learning dynamics are influenced by initial weight distributions, with small variance linked to rich learning and large variance to lazy learning.
• Biological neural circuits often have low-rank connectivity, contrasting with random initializations common in theoretical studies.

Approach:

• Investigated the impact of initial weight matrix rank on neural network learning regimes.
• Combined theoretical analysis with empirical studies, including experiments with recurrent neural networks using biologically-inspired initial connectivity.

Key Points:

• High-rank initializations generally result in less network change, indicating lazier learning.
• Low-rank initializations typically bias learning towards a richer regime with more significant changes.
• A notable exception exists: low-rank initialization can still lead to lazy learning if aligned with task and data statistics.

Conclusions:

• The effective rank of initial weights is a critical determinant of learning regimes in neural networks.
• Findings have implications for understanding the metabolic costs of neural plasticity and the risk of catastrophic forgetting in artificial and biological systems.