Consensus-Based Cooperative Algorithms for Training Over Distributed Data Sets Using Stochastic Gradients.

Area of Science:

• Distributed machine learning
• Neural network training
• Optimization algorithms

Background:

• Existing distributed neural network training methods often require true gradients or use diminishing learning rates, limiting online learning applicability.
• Lack of established principles for selecting constant learning rates hinders their effective use in distributed settings.

Purpose of the Study:

• To propose distributed algorithms for training neural networks on private datasets without requiring true gradients.
• To develop a universal model for distributed neural networks trained solely on local data using consensus mechanisms.
• To establish convergence guarantees and performance bounds for algorithms employing constant learning rates.

Main Methods:

• Utilized stochastic gradients to avoid the need for exact gradient computations.
• Introduced consensus tools to guide the distributed model towards an optimal solution using local datasets.
• Employed constant learning rates to enhance tracking ability and analyzed algorithm convergence through error dynamics.

Main Results:

• Established convergence of the proposed distributed algorithms under mild conditions, providing an upper bound for constant learning rate selection.
• Analyzed algorithm performance using mean square error (MSE), demonstrating convergence with bounded errors in the presence of gradient noise.
• Proved that MSE converges to zero in the absence of gradient noise.

Conclusions:

• The proposed distributed algorithms effectively train neural networks on private data using constant learning rates and consensus mechanisms.
• The algorithms demonstrate convergence and bounded error performance, even with gradient noise, offering a practical approach for online learning.
• The established theoretical framework provides guidance for selecting appropriate constant learning rates in distributed training scenarios.