Stochastic Mirror Descent on Overparameterized Nonlinear Models.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Deep Learning Theory

Background:

• Modern machine learning models are often overparameterized, leading to numerous solutions that perfectly fit training data.
• Understanding which solutions are found by optimization algorithms and their impact on generalization is crucial.
• Stochastic Mirror Descent (SMD) is a family of algorithms, including Stochastic Gradient Descent (SGD), used for optimization.

Purpose of the Study:

• To investigate the convergence properties of Stochastic Mirror Descent (SMD) algorithms in overparameterized learning problems.
• To analyze how initialization and learning algorithms influence the selection of interpolating solutions.
• To compare the generalization performance of different SMD variants with varying regularization strategies.

Main Methods:

• Theoretical analysis of SMD convergence for overparameterized nonlinear models with fixed step sizes.
• Empirical validation using experiments on MNIST and CIFAR-10 datasets.
• Comparison of generalization performance using l1, l2 (SGD), and l-infinity regularization.

Main Results:

• SMD algorithms approximately achieve implicit regularization by converging to solutions close to the minimum-potential interpolating solution.
• Empirical results on MNIST and CIFAR-10 confirm theoretical predictions of SMD convergence behavior.
• Experiments reveal that l-infinity regularization consistently outperforms SGD (l2) and l1 regularization in terms of generalization on CIFAR-10.

Conclusions:

• SMD algorithms exhibit implicit regularization in sufficiently overparameterized nonlinear models.
• The choice of regularization within SMD significantly impacts generalization performance, with l-infinity showing superior results.
• Findings provide insights into the behavior of optimization algorithms in deep learning and guide the selection of effective regularization techniques.