Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Weight Quantization

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Artificial Neural Networks (ANNs) face sustainability challenges due to high energy consumption.
• Spiking Neural Networks (SNNs) offer a more energy-efficient alternative using sparse, binary spikes.
• Network quantization is a technique to reduce memory and energy usage by compressing weights.

Purpose of the Study:

• Introduce Twin Network Augmentation (TNA) for improving SNN performance and enabling network compression.
• Evaluate TNA's effectiveness on benchmark datasets and its application to ternary weight precision.
• Compare TNA against traditional knowledge distillation methods.

Main Methods:

• Co-training an SNN with an identical twin SNN.
• Optimizing both networks to minimize cross-entropy losses and output logit MSE.
• Applying quantization to reduce SNN weights to ternary precision.

Main Results:

• TNA significantly enhances SNN classification performance across diverse datasets.
• TNA facilitates effective compression of SNNs to ternary weight precision.
• TNA outperforms traditional knowledge distillation and achieves state-of-the-art results on CIFAR-10, CIFAR-100, and CIFAR-10-DVS.

Conclusions:

• TNA effectively bridges the performance gap between SNNs and ANNs.
• TNA is a promising method for improving SNN efficiency and performance.
• Further research into TNA for different architectures and datasets is warranted.