Distributed Heuristic Adaptive Neural Networks With Variance Reduction in Switching Graphs

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Training neural networks with massive or privacy-sensitive data presents significant challenges.
• Existing distributed training methods struggle with dynamic communication topologies.

Purpose of the Study:

• To develop a distributed adaptive training method for neural networks operating in switching communication graphs.
• To address challenges posed by massive data and privacy concerns in distributed learning.

Main Methods:

• Utilized stochastic variance reduced gradient (SVRG) for efficient neural network training.
• Proposed a heuristic adaptive consensus algorithm to dynamically adjust agent communication weights.
• Ensured convergence through adaptive weighting based on agent performance.

Main Results:

• Demonstrated convergence of all agents to the optimum in switching communication graphs.
• Showcased reduced iteration requirements for achieving optimal performance.
• Confirmed that SVRG effectively minimizes stochastic gradient fluctuations with minimal computational overhead.

Conclusions:

• The proposed distributed heuristic adaptive neural networks offer an efficient solution for training in dynamic environments.
• The method ensures reliable convergence and improved performance, suitable for large-scale and privacy-preserving applications.