Two-layer accumulated quantized compression for communication-efficient federated learning: TLAQC.

Area of Science:

• Machine Learning
• Distributed Systems
• Data Compression

Background:

• Federated learning (FL) enables collaborative machine learning without centralizing data.
• High communication costs due to frequent gradient transfers are a major bottleneck in FL, especially for deep models.
• Existing FL methods struggle to balance communication efficiency and model accuracy.

Purpose of the Study:

• To propose a novel algorithm, Two-Layer Accumulated Quantized Compression (TLAQC), to significantly reduce communication costs in federated learning.
• To enhance the efficiency of federated learning by minimizing both individual communication overhead and the number of global communication rounds.
• To mitigate precision loss inherent in communication compression techniques.

Main Methods:

• Introduced Revised Quantized Stochastic Gradient Descent (RQSGD) with zero-value correction to reduce ineffective quantization and minimize average quantization errors.
• Implemented an adaptive threshold and parameter self-inspection mechanism to decrease the frequency of gradient information uploads.
• Employed a two-layer accumulation strategy to compensate for gradient knowledge loss by accumulating quantization errors and retained weight deltas.

Main Results:

• RQSGD achieved an ineffective quantization incidence of 0.003% and reduced average quantization error to 1.6 × 10⁻².
• TLAQC compressed uploaded traffic to 6.73% of full-precision FedAVG.
• TLAQC increased model accuracy by 1.25% compared to full-precision FedAVG.

Conclusions:

• TLAQC effectively reduces communication costs in federated learning through advanced quantization and accumulation techniques.
• The proposed RQSGD method significantly minimizes quantization errors and ineffective quantization.
• TLAQC offers a promising solution for efficient federated learning, improving accuracy while drastically reducing communication overhead.