Improved fractional-order gradient descent method based on multilayer perceptron

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Fractional-order gradient descent (FOGD) is used in Artificial Neural Networks (ANNs), but typically only applied to the loss function.
• Extending FOGD to hidden layers offers greater flexibility and potential for novel deep learning optimizers.

Purpose of the Study:

• To propose an improved fractional-order gradient descent (IFOGD) method applicable to the hidden layers of Multilayer Perceptrons (MLPs).
• To develop a fractional-order Autograd (FOAutograd) system within PyTorch for enhanced deep learning.
• To validate the performance and stability of the IFOGD method in various tasks.

Main Methods:

• Developed a fractional matrix differentiation algorithm and solver for MLP hidden layers.
• Addressed backpropagation divergence issues caused by absolute value functions.
• Implemented FOAutograd by reconstructing PyTorch's Linear layer and Mean Squared Error Loss module.
• Integrated FOAutograd with adaptive optimizers for fractional-order gradient updates.

Main Results:

• The proposed IFOGD method can be successfully applied within the hidden layers of ANNs.
• The method ensures stable loss function values, avoiding divergence.
• Simulation experiments and time series prediction tasks showed competitive performance of IFOGD.

Conclusions:

• The IFOGD method provides a robust and flexible approach to incorporating fractional-order calculus into deep learning.
• This work lays the foundation for a new family of fractional-order deep learning optimizers.
• IFOGD demonstrates significant potential for improving ANN performance in various applications.