Adaptive Batch Size Time Evolving Stochastic Gradient Descent for Federated Learning.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization Algorithms

Background:

• Variance reduction techniques enhance Stochastic Gradient Descent (SGD) in centralized settings.
• Federated learning (FL) faces challenges like mega-batch sizing, gradient noise, and statistical heterogeneity when applying variance reduction.

Purpose of the Study:

• To propose a lightweight algorithm, FedATEVR, addressing variance reduction issues in federated learning.
• To improve the efficiency and accuracy of federated learning systems.

Main Methods:

• Developed an adaptive batch size scheme using historical gradient information for clients.
• Introduced a time-evolving variance reduction gradient estimator adjusting weights based on gradient differences.
• The algorithm integrates global and local gradient information to stabilize mega-batch sizing.

Main Results:

• Theoretically proved linear speedup of O(1/sqrt(SKT)) for non-convex objectives with partial client participation.
• Empirically demonstrated superior test accuracy compared to baseline methods.
• Significantly reduced the number of communication rounds required for convergence.

Conclusions:

• FedATEVR effectively tackles key challenges in applying variance reduction to federated learning.
• The proposed method offers a practical solution for accelerating federated SGD and reducing computational costs.
• Achieved substantial improvements in both accuracy and communication efficiency in federated learning scenarios.