Advancing Spiking Neural Networks Toward Deep Residual Learning

Area of Science:

• Neuromorphic Computing
• Artificial Intelligence
• Deep Learning

Background:

• Spiking neural networks (SNNs) show promise for efficient AI but suffer from limited capacity and representation power.
• Existing deep learning techniques like residual learning are not directly applicable to SNNs, leading to degradation issues.
• Information flow and gradient propagation are significant challenges in training deep SNNs.

Purpose of the Study:

• To address the limitations of current SNNs, particularly capacity and representation power.
• To investigate the effectiveness of residual learning principles in SNN architectures.
• To propose a novel SNN-specific residual architecture that enables training of significantly deeper networks.

Main Methods:

• Developed MS-ResNet, a novel SNN-oriented residual architecture with membrane-based shortcut pathways.
• Applied block dynamical isometry theory to ensure gradient norm equality and network stability.
• Trained and evaluated MS-ResNet on benchmark datasets including CIFAR-10 and ImageNet.

Main Results:

• Successfully trained SNNs to unprecedented depths (e.g., 482 layers on CIFAR-10, 104 layers on ImageNet) without degradation.
• MS-ResNet104 achieved state-of-the-art 76.02% accuracy on ImageNet for directly trained SNNs.
• Demonstrated high energy efficiency, requiring only one spike per neuron on average for classification.

Conclusions:

• MS-ResNet effectively resolves degradation issues in deep SNNs by incorporating specialized residual pathways.
• The proposed architecture significantly enhances the performance and scalability of directly trained SNNs.
• These findings pave the way for more powerful and efficient neuromorphic computing applications.