Improving the efficiency of RMSProp optimizer by utilizing Nestrove in deep learning.

Area of Science:

• Deep Learning Optimization
• Machine Learning Algorithms
• Neural Network Training

Background:

• Deep Learning (DL) performance is often enhanced by parameter tuning like Transfer Learning, Data Augmentation, Dropout, and Batch Normalization.
• Selecting optimal optimizers and architectures is crucial for achieving peak model performance in DL.
• Optimization algorithms play a pivotal role in efficiently training DL models and minimizing loss functions.

Purpose of the Study:

• To propose a modified optimization algorithm, NRMSProp, designed to accelerate convergence in Deep Learning.
• To enhance the Root Mean Squared Propagation (RMSProp) optimizer by incorporating Nesterov Accelerated Gradient (NAG) principles.
• To achieve faster minimization of the loss function compared to the original RMSProp optimizer without increasing computational complexity.

Main Methods:

• A novel optimization algorithm, NRMSProp, is developed by modifying the existing RMSProp algorithm.
• NRMSProp integrates Nesterov Accelerated Gradient (NAG) to leverage gradient history and adapt learning rates.
• Experiments were conducted using deep Convolutional Neural Networks (CNNs) on various datasets, comparing NRMSProp against RMSProp and Adam.

Main Results:

• NRMSProp demonstrated effective performance and achieved high accuracy, reaching up to 0.97 in many test cases.
• The proposed NRMSProp algorithm showed superior convergence speed compared to the original RMSProp optimizer.
• NRMSProp achieved comparable or better results than the Adam optimizer without increased algorithmic complexity, memory, or time.

Conclusions:

• NRMSProp offers an effective and efficient optimization method for Deep Learning, outperforming RMSProp and Adam in convergence speed and accuracy.
• The integration of NAG principles into RMSProp results in a more robust and faster optimization process for DL models.
• NRMSProp presents a valuable advancement in DL optimization, providing significant performance gains with minimal computational overhead.