The importance of better models in stochastic optimization

Area of Science:

• Optimization Theory
• Machine Learning

Background:

• Standard stochastic optimization methods are often unstable and sensitive to parameter choices.
• These methods struggle with diverse objective functions, limiting their applicability.

Purpose of the Study:

• To develop more robust stochastic optimization and learning models.
• To enhance stability and convergence across a wider range of objective functions.

Main Methods:

• Introduced a new family of models termed 'aprox' for stochastic optimization.
• Extended analysis from convex to weakly convex objectives, including those common in machine learning.
• Conducted experimental evaluations of convergence time and parameter sensitivity.

Main Results:

• Demonstrated that appropriately accurate models ('aprox' family) ensure stability and provable convergence.
• Showcased that even modeling nonnegativity of the objective is sufficient for stability.
• Extended robustness guarantees to weakly convex objectives, relevant for modern machine learning.

Conclusions:

• Robust modeling is crucial for stable and efficient stochastic optimization.
• The 'aprox' family offers improved performance and reliability over standard methods.
• The findings have implications for developing more dependable machine learning algorithms.