Enhancing deep neural network training efficiency and performance through linear prediction.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) show great success but face training challenges.
• Parameter dynamics during DNN training offer potential for optimization.
• Existing training methods may not fully exploit parameter behavior.

Purpose of the Study:

• To propose a novel method for optimizing DNN training effectiveness and performance.
• To leverage parameter prediction for improved training efficiency.
• To enhance DNN performance through noise injection from prediction errors.

Main Methods:

• Developed a Parameters Linear Prediction (PLP) method for DNNs.
• Observed and utilized parameter (weights and bias) change rules during training.
• Incorporated noise injection via prediction errors to boost performance.

Main Results:

• The PLP method achieved an approximate 1% accuracy increase over Stochastic Gradient Descent (SGD).
• Top-1/top-5 error was reduced by approximately 0.01.
• Demonstrated stable performance across various hyperparameter settings.

Conclusions:

• The proposed Parameters Linear Prediction method effectively enhances DNN training efficiency and performance.
• PLP offers a viable alternative to traditional methods like SGD.
• The method shows robustness and consistent results, validating its practical utility.