Accelerating gradient descent and Adam via fractional gradients.

Area of Science:

• Optimization Algorithms
• Fractional Calculus
• Scientific Machine Learning

Background:

• Traditional optimization algorithms rely on integer-order gradients.
• Generalizing gradients to fractional orders offers potential for improved performance.
• Existing methods may struggle with ill-conditioned problems and complex objective functions.

Purpose of the Study:

• To propose and implement novel fractional-order optimization algorithms.
• To generalize gradient-based optimization using Caputo fractional derivatives.
• To demonstrate the efficacy of these new algorithms in scientific machine learning.

Main Methods:

• Definition of a fractional-order gradient via Caputo fractional derivatives (Caputo fractional-based gradient).
• Development of an efficient implementation for computing the Caputo fractional-based gradient.
• Extension of Gradient Descent (GD) and Adam to Caputo fractional Gradient Descent (CfGD) and Caputo fractional Adam (CfAdam).

Main Results:

• CfGD and CfAdam show superior performance compared to GD and Adam on large-scale scientific machine learning problems.
• Demonstrated acceleration in convergence rates for both CfGD and CfAdam.
• Error bounds derived for CfGD on quadratic functions indicate mitigation of condition number dependence.

Conclusions:

• Fractional-order optimization algorithms, specifically CfGD and CfAdam, offer significant acceleration over their integer-order counterparts.
• These novel methods are effective for challenging optimization tasks in scientific machine learning.
• The Caputo fractional-based gradient provides a powerful generalization for optimization.