Why Does Large Batch Training Result in Poor Generalization? A Comprehensive Explanation and a Better Strategy from...

Area of Science:

• Optimization Methods
• Machine Learning

Background:

• Nonconvex optimization problems are challenging to solve.
• Standard gradient descent methods can get stuck in local minima.

Purpose of the Study:

• To present a comprehensive framework of search methods for nonconvex optimization.
• To apply this framework to neural network training.
• To explain the degradation of generalization performance in large batch training.

Main Methods:

• Developing a framework incorporating simulated annealing and batch training.
• Gradually decreasing randomness in gradient descent.
• Implementing an approach for gradually increasing batch size during training.

Main Results:

• The proposed framework effectively solves nonconvex optimization problems.
• The formulation is directly applicable to neural network training.
• An effective method for gradually increasing batch size was developed.

Conclusions:

• The framework offers an effective approach for neural network training.
• The study clarifies the reasons behind generalization performance degradation with large batch sizes.