Accelerating deep neural network training with inconsistent stochastic gradient descent.

Area of Science:

• Machine Learning
• Deep Learning

Background:

• Stochastic Gradient Descent (SGD) treats all batches equally during Convolutional Neural Network (CNN) training.
• Gradient variance from sampling bias and image differences causes unequal training dynamics across batches.

Purpose of the Study:

• To introduce Inconsistent Stochastic Gradient Descent (ISGD), a new training strategy for SGD.
• To address the issue of varying training dynamics caused by gradient variance in CNNs.

Main Methods:

• ISGD dynamically adjusts training effort based on batch loss.
• It models training as a stochastic process to reduce mean batch loss.
• A dynamic upper control limit identifies high-loss batches for focused updates, with constraints to prevent drastic parameter changes.

Main Results:

• ISGD accelerates training by dedicating more updates to difficult batches.
• The method is computationally efficient and requires no auxiliary memory.
• Empirical evaluations on real-world datasets show promising performance.

Conclusions:

• ISGD offers an effective and efficient strategy for improving CNN training.
• The inconsistent training approach dynamically adapts to loss variations.
• This method enhances learning by focusing computational effort where it's most needed.