Sign-Entropy Regularization for Personalized Federated Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Personalized Federated Learning (PFL) addresses challenges of training client-specific models on heterogeneous distributed data.
• Existing PFL methods often struggle with optimization stability and personalization effectiveness due to data heterogeneity.

Purpose of the Study:

• To introduce Sign-Entropy Regularization (SER), a novel technique to improve personalization and stability in Federated Learning.
• To enhance client-specific model training by penalizing excessive directional variability in local optimization.

Main Methods:

• Developed a novel entropy-based regularization technique, Sign-Entropy Regularization (SER), inspired by Descartes' Rule of Signs.
• Defined a differentiable sign-entropy objective over gradient sign distributions and integrated it into FedAvg and FedProx.
• Applied SER efficiently post hoc per local round without modifying communication protocols.

Main Results:

• SER significantly improved both average and worst-case client accuracy across FEMNIST, Shakespeare, and CIFAR-10 datasets.
• Demonstrated reduced variance across clients, accelerated convergence, and smoothed local loss surfaces.
• Outperformed state-of-the-art personalization methods like Ditto and pFedMe in comparative evaluations.

Conclusions:

• Sign-Entropy Regularization offers a scalable and orthogonal mechanism for enhancing personalization in Federated Learning.
• SER stabilizes learning dynamics through information-theoretic and geometric regularization, applicable to resource-constrained settings.
• The method shows potential for trajectory-based regularization and hybrid entropy-guided optimization in federated learning.