Research on a learning rate with energy index in deep learning

Area of Science:

• Deep Learning
• Machine Learning Optimization

Background:

• Stochastic Gradient Descent (SGD) is a primary optimization algorithm in deep learning.
• SGD's effectiveness is highly dependent on adaptive learning rate tuning.
• Current methods often require manual or complex learning rate adjustments.

Purpose of the Study:

• To propose a novel Energy Index based Optimization Method (EIOM) for automatic learning rate adjustment in backpropagation.
• To enhance the performance of deep learning models by optimizing feature updates based on frequency.

Main Methods:

• Defined an energy neuron model to quantify feature frequency.
• Designed an energy index to represent feature importance.
• Implemented EIOM to dynamically adjust learning rates as a function of the energy index.
• Evaluated EIOM across logistic regression, multilayer perceptron, and convolutional neural network models.

Main Results:

• The proposed EIOM demonstrated promising performance in empirical evaluations.
• EIOM showed competitive or superior results compared to other optimization algorithms.
• Feature update extents were successfully differentiated based on feature frequency.

Conclusions:

• EIOM offers an effective, automated approach to learning rate optimization in deep learning.
• The energy index provides a robust mechanism for prioritizing feature updates.
• This method has the potential to improve the efficiency and accuracy of deep learning training.