The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Spiking neural networks (SNNs) are crucial for brain information processing, but understanding their connectivity-function relationship is limited.
• Current models of SNNs have rudimentary functional capabilities, hindering neuroscience research and neuromorphic hardware development.
• Training artificial neural networks using gradient descent is effective, but applying it to SNNs is challenging due to non-differentiable spike functions.

Purpose of the Study:

• To systematically investigate how surrogate gradient design parameters influence learning performance in SNNs.
• To provide insights into optimizing surrogate gradient methods for SNN training.
• To guide the development of functional SNN models and efficient neuromorphic systems.

Main Methods:

• Numerical simulations were used to evaluate surrogate gradient learning across various classification tasks.
• The study systematically analyzed the impact of essential surrogate gradient design parameters, particularly the derivative's scale.
• Surrogate gradients were combined with activity regularization techniques to assess performance under sparse activity conditions.

Main Results:

• Surrogate gradient learning demonstrates robustness to variations in the shape of surrogate derivatives.
• The scale of the surrogate derivative significantly impacts learning performance in SNNs.
• Combining surrogate gradients with activity regularization enables robust information processing in SNNs at the sparse activity limit.

Conclusions:

• Surrogate gradient learning is a remarkably robust method for training SNNs.
• The scale of the surrogate derivative is a critical parameter for optimizing SNN performance.
• This research offers a practical guide for modeling functional SNNs and advancing neuromorphic computing.