Streaming Batch Eigenupdates for Hardware Neural Networks.

Area of Science:

• Computational neuroscience
• Nanotechnology for AI
• Machine learning hardware acceleration

Background:

• Neural networks using nanodevices (memristors, phase change memory, flash) offer higher energy efficiency and density than traditional GPUs/CPUs.
• Training acceleration is possible, but space complexity of stochastic gradient descent (SGD) limits scalability.
• SGD space complexity grows quadratically with network size, posing a significant bottleneck.

Purpose of the Study:

• To reduce the space complexity inherent in training large neural networks.
• To enable more efficient training of AI models on resource-constrained nanodevice hardware.
• To improve the area, time, and energy efficiency of neural network training.

Main Methods:

• Implementation of low-rank approximations for stochastic gradient descent (SGD).
• Integration of streaming methods to manage data and computations efficiently.
• Development of the streaming batch eigenupdate (SBE) algorithm and architecture.

Main Results:

• Achieved significant reductions in memory and computational overhead.
• Demonstrated potential for substantial improvements in training area, time, and energy efficiency.
• Overcame the quadratic space complexity limitation of traditional SGD.

Conclusions:

• The streaming batch eigenupdate (SBE) approach effectively reduces space complexity for training neural networks on nanodevices.
• This method paves the way for more scalable and efficient AI hardware.
• SBE offers a promising solution for energy-efficient and high-density AI training.