Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype.

Area of Science:

• Neuroscience
• Computer Science
• Hardware Engineering

Background:

• Deep learning models typically require substantial memory, posing challenges for energy-efficient hardware.
• Existing memory-saving techniques are often applied post-training, limiting their use in on-hardware training scenarios.
• Energy-efficient hardware often has strict memory limitations, hindering the direct implementation of complex neural networks.

Purpose of the Study:

• To introduce and evaluate the Deep Rewiring (DEEP R) training algorithm for on-hardware neural network training.
• To demonstrate the feasibility of training deep neural networks within severe memory constraints on specialized hardware.
• To assess the performance and efficiency of DEEP R on the SpiNNaker 2 prototype system.

Main Methods:

• Implemented the Deep Rewiring (DEEP R) algorithm for continuous network rewiring during training.
• Trained a deep neural network on a SpiNNaker 2 prototype chip with limited local memory (64 KB per core).
• Maintained extremely sparse connectivity (1.3% active connections) throughout the training process.

Main Results:

• Achieved 96.6% classification accuracy on the MNIST dataset with the highly sparse network.
• Demonstrated excellent scaling of computation time, memory usage, and energy efficiency across SpiNNaker's multi-processor system.
• Observed a two-orders-of-magnitude improvement in power and energy consumption compared to x86 CPU implementations.

Conclusions:

• Deep Rewiring (DEEP R) is an effective training algorithm for deploying deep neural networks on memory-constrained, energy-efficient hardware.
• The SpiNNaker 2 prototype, utilizing DEEP R, offers a highly power-efficient platform for neural network training.
• This approach significantly advances the potential for on-device AI and neuromorphic computing.