ESTSformer: Efficient spatio-temporal spiking transformer.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Spiking Neural Networks (SNNs) offer efficient, event-driven computation, contrasting with traditional Artificial Neural Networks (ANNs).
• Transformers are increasingly vital in AI, but integrating them with SNNs requires overcoming challenges in leveraging spatio-temporal dynamics.
• Existing spiking spatio-temporal attention mechanisms face limitations, particularly high computational and storage demands with increasing time steps.

Purpose of the Study:

• To address the limitations of vanilla spatiotemporal self-attention (STSA) in SNNs.
• To propose an efficient spatio-temporal self-attention (ESTSA) mechanism for improved SNN performance.
• To develop an efficient spiking transformer architecture (ESTSformer) for purely spike-driven computation.

Main Methods:

• Analyzed drawbacks of vanilla STSA, focusing on computational and storage escalations.
• Developed ESTSA by partitioning attention heads for distinct temporal and spatial feature extraction.
• Constructed ESTSformer by integrating ESTSA and modifying residual connections for spike-driven processing.

Main Results:

• ESTSA significantly reduces computational and storage overhead compared to vanilla STSA.
• ESTSformer demonstrates superior performance and efficiency on both neuromorphic and static datasets.
• The proposed architecture achieves better results than existing advanced methods.

Conclusions:

• The ESTSformer provides an efficient and high-performing solution for integrating Transformers with SNNs.
• ESTSA is a key innovation enabling reduced resource demands while maintaining effectiveness.
• This work advances the development of efficient bio-inspired AI models.