Spike-HAR++: an energy-efficient and lightweight parallel spiking transformer for event-based human action...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Neuromorphic Engineering

Background:

• Event-based cameras offer advantages for human action recognition (HAR) due to their high dynamic range, temporal resolution, power efficiency, and low latency.
• Spike Neural Networks (SNNs) are well-suited for event-based data due to their event-driven nature and lower power consumption compared to traditional neural networks.

Purpose of the Study:

• To introduce novel end-to-end Spiking Neural Networks (SNNs), Spike-HAR and Spike-HAR++, for event-based human action recognition (HAR).
• To enhance the accuracy and computational efficiency of SNNs for HAR by incorporating spiking transformer mechanisms.

Main Methods:

• Development of Spike-HAR with a spike attention branch for noise reduction and a simplified spiking self-attention transformer block for efficiency.
• Extension of Spike-HAR to Spike-HAR++ by modifying the spike attention branch for higher-dimensional feature extraction, improving classification performance.
• Comprehensive evaluation on four diverse HAR datasets: SL-Animals-DVS, N-LSA64, DVS128 Gesture, and DailyAction-DVS.

Main Results:

• Spike-HAR and Spike-HAR++ demonstrated superior performance in event-based HAR across multiple datasets.
• The proposed models achieved high accuracy with significantly low energy consumption (0.03 mJ for Spike-HAR, 0.06 mJ for Spike-HAR++).
• The models exhibit small sizes (0.7 M for Spike-HAR, 1.8 M for Spike-HAR++), indicating high efficiency.

Conclusions:

• Spike-HAR and Spike-HAR++ represent a significant advancement in SNN-based event-based HAR, offering a promising new baseline.
• The proposed models achieve state-of-the-art performance with remarkable power and computational efficiency.
• The availability of code facilitates further research and adoption within the HAR community.