Sign-Based Gradient Descent With Heterogeneous Data: Convergence and Byzantine Resilience

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Communication overhead is a key bottleneck in distributed deep neural network training.
• SignSGD with majority vote offers communication efficiency and Byzantine robustness.
• SignSGD's convergence is hindered by data heterogeneity, prevalent in federated learning.

Purpose of the Study:

• To address the non-convergence issue of SignSGD in federated learning due to data heterogeneity.
• To develop a novel gradient compression method for sign-based stochastic gradient descent.
• To establish theoretical convergence guarantees and quantify Byzantine resilience.

Main Methods:

• Derivation of a sufficient condition for the convergence of sign-based gradient descent.
• Proposal of a magnitude-driven stochastic-sign-based gradient compressor.
• Incorporation of an error-feedback mechanism to enhance learning performance.

Main Results:

• The proposed method achieves convergence in the presence of arbitrary data heterogeneity.
• The Byzantine resilience of sign-based gradient descent methods is quantified.
• Experimental validation on MNIST, CIFAR-10, and Tiny-ImageNet datasets demonstrates effectiveness.

Conclusions:

• The novel magnitude-driven compressor effectively resolves SignSGD's non-convergence issues in heterogeneous federated learning settings.
• The method maintains robustness against Byzantine attacks while improving learning performance.
• This work advances efficient and reliable distributed deep learning training.