MSFI: Multi-timescale spatio-temporal features integration in spiking neural networks

Area of Science:

• Neuromorphic Engineering
• Artificial Intelligence
• Computer Vision

Background:

• Dynamic vision sensors (DVS) capture temporal information with high resolution.
• Spiking neural networks (SNNs) process event streams but often neglect spatial dependencies.
• Existing SNNs struggle to extract complex spatio-temporal features from DVS data.

Purpose of the Study:

• To propose a novel plug-and-play module for SNNs to enhance spatio-temporal feature extraction.
• To improve the representative capabilities of SNNs when processing DVS data.
• To address the limitations of current SNNs in handling spatial information and complex temporal dynamics.

Main Methods:

• Introduction of the Multi-timescale Spatio-temporal Features Integration (MSFI) module.
• MSFI incorporates Short-term Spatio-temporal Module (SSM) and Long-term Spatio-temporal Module (LSM).
• Fusion of extracted spatio-temporal features with original spiking features within SNNs.

Main Results:

• The proposed MSFI module significantly enhances SNN performance.
• Demonstrated improvements on neuromorphic datasets (CIFAR10-DVS, DVS128 Gesture, DVS128 Gait).
• Achieved superior results on static datasets (CIFAR10/100, ImageNet) compared to baselines.

Conclusions:

• The MSFI module effectively extracts and integrates multi-timescale spatio-temporal features.
• MSFI enhances SNNs' ability to process complex data from DVS and static image sources.
• The proposed approach offers a significant advancement in SNN capabilities for vision tasks.