Auto-Spikformer: Spikformer architecture search.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience

Background:

• Spiking Neural Networks (SNNs) integrate self-attention mechanisms for advanced deep learning.
• Spikformer shows promise but may suffer from high energy consumption due to redundant architecture.
• Optimizing SNNs for energy efficiency is crucial for their practical application.

Purpose of the Study:

• To develop an efficient architecture search method for Spiking Transformers.
• To reduce the energy consumption of Spiking Neural Networks without compromising performance.
• To explore a broader search space including both architecture and inner SNN parameters.

Main Methods:

• Proposed Auto-Spikformer, a one-shot Spiking Transformer Architecture Search method.
• Utilized weight entanglement, evolutionary search, and Discrete Spiking Parameters Search (DSPS).
• Introduced a novel fitness function for Pareto optimal balancing of energy and accuracy.

Main Results:

• Auto-Spikformer subnets achieved performance comparable to Spikformer with inherited weights.
• The proposed method significantly reduced energy consumption compared to the original Spikformer.
• Auto-Spikformer outperformed Spikformer and other models (CNN, ViT) in efficiency and parameter count.

Conclusions:

• Auto-Spikformer effectively optimizes Spiking Transformer architectures for enhanced energy efficiency.
• The method demonstrates a viable approach to balance accuracy and energy consumption in SNNs.
• This research contributes to the development of more biologically plausible and efficient deep learning models.