SAD-SNN: Spatial-Activation Distillation for High-Performance Spiking Neural Networks

Area of Science:

• Artificial Intelligence
• Computer Science
• Neuroscience

Background:

• Spiking Neural Networks (SNNs) offer energy-efficient, brain-inspired alternatives to Artificial Neural Networks (ANNs).
• Current direct training methods for SNNs yield unsatisfactory performance, limiting their practical application.
• Existing knowledge transfer methods often rely on element-wise feature alignment, which can lead to precision loss.

Purpose of the Study:

• To introduce a novel learning method, Spatial-Activation Distillation for Spiking Neural Networks (SAD-SNN), for improved SNN training.
• To enable effective knowledge transfer from teacher ANNs to student SNNs using spatial-activation map alignment.
• To enhance SNN performance in energy-constrained applications like image classification and signal detection.

Main Methods:

• Developed SAD-SNN, a method that guides SNN learning using a teacher ANN model.
• Implemented spatial-activation map alignment at different resolutions between teacher and student networks.
• Introduced a direct alignment approach with a spatial-activation loss and normalized representation vectors to prevent precision loss.

Main Results:

• SAD-SNN outperformed existing SNN training methods on three image classification datasets, using both homogeneous and heterogeneous teacher ANNs.
• The method demonstrated strong generalization ability and superior performance when applied to electromagnetic signal detection.
• Significant improvements in student SNN learning were achieved with only two time steps.

Conclusions:

• SAD-SNN is a general and effective solution for improving SNN learning.
• The spatial-activation distillation approach facilitates robust knowledge transfer from ANNs to SNNs.
• The method shows promise for enhancing SNN performance across various tasks and architectures, especially in low-energy scenarios.