LDD: High-Precision Training of Deep Spiking Neural Network Transformers Guided by an Artificial Neural Network

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Large-scale Transformer models face significant computational and energy demands.
• Spiking Neural Networks (SNNs) offer a promising alternative for energy-efficient and high-speed AI.
• Directly training deep SNN Transformers is hindered by surrogate gradient inaccuracies and feature quantization.

Purpose of the Study:

• To develop a method for effectively training deep Spiking Neural Network (SNN) Transformer models.
• To address the challenges of surrogate gradient inaccuracy and feature quantization in SNN training.
• To align feature representations between Artificial Neural Networks (ANNs) and SNNs across different network abstraction levels.

Main Methods:

• Proposed a novel method, LDD (Layer-wise Distillation for Deep SNN Transformers).
• Incorporated structured feature knowledge from ANNs to guide SNN training.
• Designed layer-wise distillation losses to mitigate surrogate gradient inaccuracies and preserve crucial information.

Main Results:

• Achieved state-of-the-art performance on CIFAR10 (96.1%), CIFAR100 (82.3%), and ImageNet (80.9%) datasets.
• Successfully enabled the training of the deepest SNN Transformer network to date on ImageNet.
• Demonstrated the effectiveness of LDD in aligning ANN and SNN features across abstraction levels.

Conclusions:

• LDD effectively overcomes key challenges in training deep SNN Transformers.
• The proposed method significantly improves accuracy and enables deeper SNN Transformer architectures.
• LDD paves the way for more energy-efficient and powerful AI models using SNNs.