Learning long sequences in spiking neural networks

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Spiking neural networks (SNNs) offer energy-efficient computation but lag behind Transformers in sequential tasks due to RNN limitations and training challenges.
• State space models (SSMs) have emerged as efficient alternatives to Transformers for sequence modeling.

Purpose of the Study:

• To investigate the integration of state-of-the-art SSMs with SNNs for long-range sequence modeling.
• To evaluate the performance of SSM-based SNNs against Transformers and existing SNNs.

Main Methods:

• Systematic investigation of SSM-SNN intersection for long-range sequence modeling.
• Introduction of a novel feature mixing layer to enhance SNN accuracy.
• Benchmarking against established long-range sequence modeling tasks and sequential image classification.

Main Results:

• SSM-based SNNs outperformed Transformer models on all tasks in a long-range sequence modeling benchmark.
• SSM-based SNNs achieved superior performance compared to state-of-the-art SNNs with fewer parameters in sequential image classification.
• A novel feature mixing layer improved SNN accuracy, questioning prior assumptions about binary activations.

Conclusions:

• SSM-based SNNs represent a significant advancement for energy-efficient long-range sequence modeling.
• This research enables the deployment of powerful SSM architectures, like large language models, on neuromorphic hardware.
• The findings open new avenues for efficient and brain-inspired AI.