Accelerating DNN Training Through Selective Localized Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Deep Neural Networks (DNNs) training demands significant computational resources, time, and energy.
• Back-propagation, a standard DNN training method, involves computationally intensive Generalized Matrix Multiply (GEMM) operations.

Purpose of the Study:

• To introduce LoCal+SGD, an algorithmic approach to accelerate DNN training.
• To reduce the computational and memory footprint of DNN training.

Main Methods:

• LoCal+SGD selectively combines localized/Hebbian learning with Stochastic Gradient Descent (SGD).
• A Learning Mode Selection Algorithm dynamically transitions layers between localized updates (1 GEMM) and gradient-based SGD updates.
• Proposes static and dynamic approaches for the selection algorithm and a weak supervision mechanism for localized learning rates.

Main Results:

• Achieved up to 1.5x improvement in end-to-end training time on 8 Convolutional Neural Networks (CNNs) across 3 datasets.
• Demonstrated minimal ~0.5% loss in Top-1 classification accuracy.
• Reduced memory footprint by eliminating the need to store layer activations for localized update layers.

Conclusions:

• LoCal+SGD offers a viable method for accelerating DNN training.
• The approach effectively balances training speed improvements with accuracy preservation.
• LoCal+SGD presents a promising direction for efficient deep learning model development.