Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Signal Processing

Background:

• Spiking neural networks (SNNs) integrated with transformers offer energy efficiency and high accuracy.
• Standard spiking self-attention (SSA) may be computationally intensive and not always necessary for sparse spike data.
• The hypothesis that both SSA and transforms like Fourier and Wavelet utilize basis functions for information processing.

Purpose of the Study:

• To propose and evaluate a novel energy-efficient spikformer architecture.
• To replace dynamic bases in SSA with fixed bases from Fourier and Wavelet transforms.
• To assess the performance of the proposed Fourier-or-Wavelet-based spikformer (FWformer) in visual classification.

Main Methods:

• Developed the Fourier-or-Wavelet-based spikformer (FWformer) by substituting standard SSA with spike-form Fourier transform, wavelet transform, or their combinations.
• Utilized fixed triangular or wavelet bases for information transformation instead of dynamic bases.
• Validated the FWformer on visual classification tasks using static image and event-based video datasets.

Main Results:

• FWformer achieved comparable or higher accuracies (0.4%-1.5%) compared to standard spikformer.
• Demonstrated significant improvements in running speed: 9%-51% for training and 19%-70% for inference.
• Reported reduced theoretical energy consumption (20%-25%) and GPU memory usage (4%-26%).

Conclusions:

• The FWformer offers a more efficient alternative to standard spikformer by leveraging fixed bases from Fourier and Wavelet transforms.
• This approach successfully balances accuracy with substantial gains in speed and reductions in computational cost.
• The findings suggest that refining transformers inspired by biological signals (spikes) and information theory (transforms) is a promising research direction.