Accelerating Minibatch Stochastic Gradient Descent Using Typicality Sampling.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) have advanced rapidly in AI fields.
• Minibatch stochastic gradient descent (SGD) is a popular training method for DNNs.
• Conventional minibatch SGD exhibits slow convergence due to noisy gradient approximations.

Purpose of the Study:

• To develop a more efficient batch selection method for training deep networks.
• To reduce gradient estimation errors in minibatch SGD.
• To enhance the convergence rate of deep learning models.

Main Methods:

• A novel batch selection strategy based on typicality sampling was developed.
• The proposed typical batch SGD algorithm was analyzed for its convergence rate.
• Convergence properties were compared between typical batch SGD and conventional minibatch SGD.

Main Results:

• The typicality sampling batch selection method effectively reduces gradient estimation error.
• The typical batch SGD algorithm demonstrates improved convergence properties compared to standard minibatch SGD.
• The proposed batch selection strategy benefits more complex minibatch SGD variants.

Conclusions:

• Typical batch SGD offers a more efficient approach to training deep neural networks.
• The developed batch selection method enhances convergence speed and accuracy.
• This strategy can be integrated with advanced minibatch SGD variants for further performance gains.