FedTP: Federated Learning by Transformer Personalization.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Distributed Systems

Background:

• Federated learning enables privacy-preserving collaborative model training.
• Personalized federated learning addresses data heterogeneity across clients.
• Transformers are increasingly applied to federated learning, but their self-attention mechanisms remain understudied.

Purpose of the Study:

• Investigate the impact of federated learning algorithms on transformer self-attention.
• Propose a novel framework, FedTP, to enhance transformer performance in heterogeneous federated settings.
• Develop a learn-to-personalize mechanism for scalable and generalizable personalized self-attention.

Main Methods:

• Analyzed the effect of federated averaging (FedAvg) on self-attention under data heterogeneity.
• Introduced FedTP, a transformer-based federated learning framework with client-specific self-attention.
• Implemented a learn-to-personalize mechanism using a server-side hypernetwork for personalized projection matrices.
• Derived a generalization bound for FedTP with the learn-to-personalize approach.

Main Results:

• Federated averaging (FedAvg) negatively impacts self-attention with heterogeneous data.
• FedTP effectively learns personalized self-attention for each client.
• The learn-to-personalize mechanism enhances cooperation, scalability, and generalization.
• FedTP achieves state-of-the-art performance in non-IID scenarios.

Conclusions:

• Standard federated learning algorithms can hinder transformer capabilities in heterogeneous environments.
• FedTP offers a robust solution for personalized federated learning with transformers.
• The proposed learn-to-personalize mechanism is crucial for FedTP's superior performance and scalability.