Fractional Gradient Descent With Matrix Stepsizes for Non-Convex Optimization.

Area of Science:

• Optimization Theory
• Machine Learning
• Non-convex Optimization

Background:

• Fractional derivatives generalize integer-order derivatives, relevant for optimization algorithms.
• Existing convergence analysis for fractional gradient descent (FGD) is limited in scope and applicable settings.
• Non-convex optimization problems are prevalent in machine learning and require robust algorithms.

Purpose of the Study:

• Establish convergence guarantees for FGD on a broader class of non-convex functions (matrix-smooth functions).
• Propose novel stochastic fractional descent algorithms (CFGD) with matrix-valued stepsizes.
• Analyze convergence in both single-node and distributed settings for matrix-smooth objectives.

Main Methods:

• Leveraging matrix smoothness properties to prove convergence and accelerate FGD iterates.
• Developing two novel stochastic fractional descent algorithms (CFGD).
• Incorporating matrix-valued stepsizes to minimize matrix-smooth non-convex objectives.

Main Results:

• Established convergence guarantees for FGD on matrix-smooth non-convex functions.
• Demonstrated that matrix stepsizes lead to faster convergence than scalar stepsizes by better capturing objective structure.
• Showcased the effectiveness of matrix stepsizes in leveraging model structure.

Conclusions:

• This work provides the first convergence analysis of FGD for matrix-smooth non-convex functions.
• Introduced novel CFGD algorithms that outperform traditional methods in distributed settings.
• Highlighted the significance of matrix stepsizes for efficient optimization in federated/distributed learning.