Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Federated learning (FL) typically restricts clients to a single global model, limiting personalization.
• CLIP-like Vision-Language Models (VLMs) offer robust representation learning via prompt learning.
• Existing federated prompt learning methods inherit limitations of the traditional FL paradigm.

Purpose of the Study:

• To propose a novel federated prompt learning framework, Personalized Federated Mixture of Adaptive Prompts (pFedMoAP).
• To address the limitations of traditional FL in handling lightweight prompts for VLMs.
• To enhance personalization in federated learning by enabling clients to leverage multiple pre-aggregated prompts.

Main Methods:

• Introduced pFedMoAP, a federated learning framework utilizing a Mixture of Experts (MoE) approach for prompt learning.
• Facilitated clients to download multiple pre-aggregated prompts as fixed non-local experts.
• Implemented a local attention-based gating network for personalized prompt adaptation and enhanced text feature generation.

Main Results:

• Demonstrated the efficacy of pFedMoAP across 9 diverse datasets under various federated settings.
• Showcased improved alignment between local image data and enhanced text features through adaptive prompt experts.
• Validated the benefits of personalization in federated prompt learning.

Conclusions:

• pFedMoAP effectively personalizes federated prompt learning by integrating non-local experts within a Mixture of Experts framework.
• The proposed method enhances VLM representation learning in federated settings, outperforming traditional approaches.
• The framework offers a flexible and efficient solution for personalized federated learning with lightweight prompts.