Enhancing SNN-based spatio-temporal learning: A benchmark dataset and Cross-Modality Attention model.

Area of Science:

• Neuromorphic computing
• Artificial intelligence
• Computer vision

Background:

• Spiking Neural Networks (SNNs) offer low power consumption and brain-inspired processing.
• Existing neuromorphic datasets often lack sufficient temporal correlation for SNNs.
• Integrating event and frame data can provide richer spatio-temporal information.

Purpose of the Study:

• Introduce DVS-SLR, a novel neuromorphic dataset with high temporal correlation.
• Address the underexplored area of SNN-based cross-modality fusion.
• Develop and evaluate a fusion method to leverage dual-modal data for SNNs.

Main Methods:

• Developed the DVS-SLR dataset featuring high temporal correlation and dual-modal (event and frame) data.
• Proposed a Cross-Modality Attention (CMA) based fusion method for SNNs.
• Utilized CMA to learn and allocate spatio-temporal attention scores across event and frame modalities.

Main Results:

• The DVS-SLR dataset demonstrates higher temporal correlation, larger scale, and greater scenario diversity than existing datasets.
• The CMA fusion method enhances SNN recognition accuracy.
• The proposed approach ensures robustness across diverse scenarios.

Conclusions:

• The DVS-SLR dataset effectively enables SNNs to exploit their spatio-temporal capabilities.
• The CMA method successfully fuses event and frame data, improving SNN performance and robustness.
• This work advances SNNs by providing a suitable dataset and an effective fusion technique.