Online Learning for DNN Training: A Stochastic Block Adaptive Gradient Algorithm.

Area of Science:

• Machine Learning
• Deep Learning
• Optimization Algorithms

Background:

• Deep neural networks (DNNs) are powerful but computationally expensive to train.
• Traditional training methods often require full gradient computation, hindering efficiency for complex DNNs.

Purpose of the Study:

• To present a computationally efficient algorithm for training deep neural networks.
• To reduce the prohibitive training costs associated with full gradient computation in complex DNNs.

Main Methods:

• Introduced Stochastic Block Adaptive Gradient (SBAG) online training algorithm.
• Utilized stochastic block coordinate descent and adaptive learning rates within each iteration.
• Proved a regret bound of O(sqrt(T)) for the SBAG algorithm, where T is the time horizon.

Main Results:

• SBAG demonstrated improved training speed compared to existing methods.
• The algorithm showed enhanced generalized ability on benchmark datasets.
• Experiments on ResNet-34 and DenseNet-121 on CIFAR-10 validated SBAG's effectiveness.

Conclusions:

• SBAG offers a viable solution to reduce the computational cost of training DNNs.
• The algorithm achieves a favorable regret bound, indicating efficient online learning.
• SBAG presents a promising approach for faster and more generalized DNN training.