DisSAGD: A Distributed Parameter Update Scheme Based on Variance Reduction.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Stochastic Gradient Descent (SGD) often suffers from slow convergence and instability due to high variance in sample gradient estimates.
• Traditional distributed SGD methods may not fully address these issues, impacting training efficiency.

Purpose of the Study:

• To propose a novel distributed SGD algorithm, DisSAGD, that enhances convergence speed and stability.
• To reduce the impact of gradient variance in distributed machine learning training.

Main Methods:

• Developed DisSAGD, a distributed SGD algorithm incorporating variance reduction techniques.
• Implemented DisSAGD on distributed clusters using asynchronous communication for parameter sharing.
• Introduced adaptive learning rate and sampling strategies to mitigate update lag and optimize parameter distribution.

Main Results:

• DisSAGD effectively reduces gradient estimation errors by utilizing historical gradient variance.
• Experiments show DisSAGD significantly decreases waiting times in loop iterations compared to traditional methods.
• The proposed method achieves notable speed increases in distributed cluster training.

Conclusions:

• DisSAGD offers a more stable and faster convergence solution for distributed machine learning.
• The algorithm's variance reduction and adaptive strategies contribute to improved training performance.
• DisSAGD demonstrates practical benefits for large-scale machine learning model training in distributed environments.