Lightweight spiking transformer towards neurodynamic integration framework.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Neuromorphic Engineering

Background:

• Spiking Neural Networks (SNNs) offer energy efficiency but struggle with complex dependencies.
• Transformer architectures excel at modeling global dependencies but are computationally intensive.
• Integrating Transformers into SNNs is challenging due to computational costs and the spiking paradigm.

Purpose of the Study:

• To propose a unified neuro-dynamic spiking Transformer model, the Lightweight Spiking Transformer (LST).
• To enhance computational efficiency and hardware-friendliness in spiking Transformer models.
• To improve performance on visual processing tasks using SNNs.

Main Methods:

• Developed a Spiking Visual Tokenizer (SVT) for efficient downsampling.
• Introduced a Spiking Lambda Attention (SLA) mechanism to avoid explicit attention matrices.
• Incorporated a Dual-Threshold Adaptive LIF (DTA-LIF) neuron for enhanced signal representation.

Main Results:

• Achieved high top-1 accuracy on benchmark datasets: Tiny-Imagenet (61.15%), CIFAR10 (96.11%), CIFAR100 (79.24%), and DVS128 Gesture (98.4%).
• Reduced parameter count compared to state-of-the-art Spiking Transformer models.
• Demonstrated a balance between performance and energy efficiency.

Conclusions:

• The proposed LST framework offers an efficient and hardware-friendly design for visual processing using SNNs.
• LST provides a promising solution for high-performance, low-power visual tasks.
• The model effectively integrates Transformer capabilities within the SNN paradigm.