Tempered fractional gradient descent: Theory, algorithms, and robust learning applications.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Fractional Calculus

Background:

• Traditional gradient descent methods exhibit slow convergence and oscillations in high-dimensional, noisy data landscapes.
• Existing optimizers struggle with complex optimization problems, necessitating more robust approaches.

Purpose of the Study:

• Introduce Tempered Fractional Gradient Descent (TFGD), a novel optimization framework.
• Enhance gradient-based learning by integrating fractional calculus and exponential tempering.
• Address limitations of traditional methods in convergence and stability.

Main Methods:

• Developed TFGD by incorporating a tempered memory mechanism with fractional coefficients and exponential decay.
• Analyzed theoretical convergence guarantees for convex and stochastic settings.
• Validated TFGD's performance empirically on diverse benchmark datasets.

Main Results:

• TFGD achieved superior accuracy on Breast Cancer Wisconsin (98.25%) and MNIST (99.1%) compared to SGD and Adam.
• Demonstrated 2x faster convergence than SGD in medical classification and smoother optimization in non-convex settings.
• Identified optimal hyperparameter ranges (α=0.6-0.7, λ=0.3-0.5) for noisy data.

Conclusions:

• TFGD offers a robust alternative to conventional optimizers, improving convergence and stability.
• The tempered memory mechanism is effective for datasets with correlated features.
• TFGD shows significant promise for both theoretical and applied machine learning tasks.