Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Training spiking neural networks (SNNs) for complex functions is vital for understanding brain information processing and advancing neuromorphic computing.
• The binary nature of neuronal spikes hinders direct gradient-based training, necessitating alternative approaches like surrogate gradients.
• The theoretical underpinnings of surrogate gradients, while empirically successful, have remained largely unexplored.

Purpose of the Study:

• To investigate the theoretical foundation of surrogate gradients in the context of training spiking neural networks.
• To explore the relationship between surrogate gradients and established theoretical frameworks like smoothed probabilistic models and stochastic automatic differentiation.
• To provide theoretical support for the practical effectiveness and application of surrogate gradients, particularly in stochastic SNNs.

Main Methods:

• Comparison of surrogate gradients with derivatives from smoothed probabilistic models for single neurons.
• Investigation of stochastic automatic differentiation for training SNNs and its connection to surrogate gradients.
• Empirical validation of surrogate gradients in stochastic multilayer SNNs.

Main Results:

• Surrogate gradients are shown to be equivalent to derivatives from smoothed probabilistic models for single neurons.
• Stochastic automatic differentiation provides a theoretical basis for surrogate gradients in stochastic SNNs, matching the derivative of the neuronal escape noise function.
• Surrogate gradients are confirmed to be effective in stochastic multilayer SNNs, though not generally gradients of a surrogate loss.

Conclusions:

• The study establishes a theoretical foundation for surrogate gradients in stochastic spiking neural networks.
• The findings support the practical efficacy and suitability of surrogate gradients for SNNs, especially stochastic variants.
• This work validates the use of surrogate gradients and guides the selection of appropriate surrogate derivatives in SNN research and development.