On Consensus-Optimality Trade-offs in Collaborative Deep Learning.

Area of Science:

• Distributed Machine Learning
• Deep Learning
• Optimization Algorithms

Background:

• Distributed machine learning involves agents learning from private data, facing a trade-off between consensus and optimality.
• Existing methods for collaborative deep learning require exploration of consensus-optimality dynamics.

Purpose of the Study:

• To explore consensus-optimality trade-offs in distributed deep learning over fixed communication topologies.
• To propose novel algorithms that manage the balance between agent agreement and individual model performance.

Main Methods:

• Introduction of incremental consensus-based distributed stochastic gradient descent (i-CDSGD) with multiple consensus steps per SGD iteration.
• Development of generalized consensus-based distributed SGD (g-CDSGD) to cover the spectrum from full consensus to full disagreement.
• Analytical convergence proofs for both algorithms with strongly convex and non-convex objective functions, including momentum variants.

Main Results:

• Analytical convergence established for the proposed i-CDSGD and g-CDSGD algorithms under various conditions.
• Numerical experiments demonstrate significant performance improvements over existing collaborative deep learning methods.
• The g-CDSGD algorithm effectively navigates the consensus-optimality spectrum.

Conclusions:

• The proposed i-CDSGD and g-CDSGD algorithms offer effective solutions for managing consensus-optimality trade-offs in distributed deep learning.
• These algorithms provide enhanced performance and flexibility for collaborative learning scenarios.
• The findings advance the field of distributed optimization for deep learning applications.