Efficient Federated Learning Via Local Adaptive Amended Optimizer With Linear Speedup.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization Algorithms

Background:

• Adaptive optimization is successful in distributed learning but faces inefficiencies when extended to federated learning (FL).
• Challenges in FL include rugged convergence from inaccurate global gradients and client drift due to local overfitting.
• Existing adaptive optimizers struggle with the unique demands of decentralized, heterogeneous data environments in FL.

Purpose of the Study:

• To propose a novel momentum-based algorithm, FedLADA, to address the inefficiencies in federated learning optimization.
• To improve empirical training speed and mitigate heterogeneous overfitting in federated learning settings.
• To theoretically establish the convergence rate and linear speedup property of the proposed algorithm.

Main Methods:

• Developed Federated Local ADaptive Amended optimizer (FedLADA), a momentum-based algorithm combining global gradient descent with a locally amended adaptive optimizer.
• FedLADA estimates and corrects local model offsets using a momentum-like term based on global average offsets from previous rounds.
• Theoretical analysis established convergence rates for non-convex cases under partial client participation.

Main Results:

• FedLADA demonstrates improved empirical training speed and effectively mitigates heterogeneous overfitting.
• Experimental results on real-world datasets show FedLADA significantly reduces communication rounds compared to baselines.
• The proposed algorithm achieves higher accuracy than several existing federated learning optimization methods.

Conclusions:

• FedLADA offers an effective solution to the convergence and client drift issues in federated learning.
• The algorithm shows promise for accelerating training and enhancing model accuracy in decentralized learning environments.
• FedLADA provides a theoretically sound and empirically validated approach for efficient federated learning optimization.