Effect of Depth and Width on Local Minima in Deep Learning.

Area of Science:

• Machine Learning
• Deep Learning Theory

Background:

• Existing research often relies on strong overparameterization or simplification assumptions to analyze neural network properties.
• Understanding the impact of network architecture (depth and width) on the quality of local minima is crucial for effective deep learning model training.

Purpose of the Study:

• To theoretically analyze the influence of depth and width on the quality of local minima in deep nonlinear neural networks.
• To investigate whether local minima in deep neural networks can achieve performance comparable to or better than classical machine learning models.
• To relax the common assumption of strong overparameterization in theoretical analyses of neural networks.

Main Methods:

• Theoretical analysis of deep nonlinear neural networks with squared loss function.
• Mathematical proofs establishing relationships between network dimensions and local minima quality.
• Empirical validation using synthetic datasets and standard benchmarks (MNIST, CIFAR-10, SVHN).

Main Results:

• The quality of local minima improves towards the global minimum value as network depth and width increase, without simplification assumptions.
• Theoretical results demonstrate that local minima in deep neural networks are no worse than globally optimal values from classical machine learning models.
• Empirical results on synthetic and real-world datasets (MNIST, CIFAR-10, SVHN) support the theoretical findings.

Conclusions:

• Network depth and width are critical factors for improving the quality of local minima in deep learning.
• The study provides a theoretical foundation for understanding local minima in deep neural networks without requiring strong overparameterization.
• The findings suggest deep neural networks can achieve performance superior to classical models, even at non-overparameterized regimes.