Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Distributed learning trains large models across private data, but faces Byzantine robustness and communication efficiency challenges.
• Current methods require full gradients and converge to large solution neighborhoods.
• Existing Byzantine-robust methods struggle with communication efficiency and gradient compression.

Purpose of the Study:

• To develop a novel Byzantine-robust and communication-efficient stochastic distributed learning method.
• To achieve convergence to a smaller neighborhood, aligning with theoretical lower bounds.
• To address limitations of existing methods that rely on full gradient information.

Main Methods:

• Leveraging Polyak Momentum to mitigate noise from biased compressors and stochastic gradients.
• Developing a stochastic distributed learning algorithm with no batch size requirements.
• Providing theoretical analysis with tight complexity bounds for nonconvex smooth loss functions.

Main Results:

• The proposed method demonstrates Byzantine robustness under information compression.
• Achieves convergence to a smaller neighborhood compared to existing methods.
• Experimental validation on binary and image classification tasks confirms practical significance.

Conclusions:

• The novel method offers improved Byzantine robustness and communication efficiency in distributed learning.
• Polyak Momentum is key to mitigating noise and defending against Byzantine workers.
• The algorithm shows practical effectiveness and theoretical guarantees for nonconvex optimization.