Low Complexity Gradient Computation Techniques to Accelerate Deep Neural Network Training.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural network (DNN) training relies on iterative weight updates via gradient computation, typically using stochastic gradient descent (SGD).
• Reducing the computational complexity of gradient calculations offers a promising avenue for energy and time savings in DNN training.

Purpose of the Study:

• To propose novel techniques for reducing computational complexity in gradient computations for accelerating SGD-based DNN training.
• To investigate methods for skipping gradient computations without compromising DNN accuracy.
• To explore approximations for intermediate activations to reduce computational cost.

Main Methods:

• Exploiting the relationship between network confidence and weight gradient magnitude to skip computations for high-confidence inputs.
• Applying angle diversity-based approximations for intermediate activations, reducing bit precision (2-/4-/8-bit) based on angle error.
• Simulating the proposed techniques on CIFAR-10 dataset using ResNet-20.

Main Results:

• The proposed approach achieved skipping up to 75.83% of gradient computations with negligible accuracy degradation.
• Hardware implementation demonstrated up to 1.69x energy efficiency compared to other training accelerators.
• Reduced bit precision for activations effectively managed angle errors in early training epochs.

Conclusions:

• The developed techniques significantly reduce computational complexity in DNN training, leading to substantial energy and time savings.
• The proposed methods offer a practical approach to accelerate training without sacrificing model accuracy.
• Hardware implementations confirm the energy efficiency benefits of the proposed training accelerator.