Token-Level Prompt Mixture With Parameter-Free Routing for Federated Domain Generalization

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Federated Domain Generalization (FedDG) trains models on decentralized, heterogeneous data.
• Existing prompt learning methods for FedDG struggle with sample diversity, leading to performance degradation.
• Mixture of Experts (MoE) architectures offer specialization but face challenges in expert assignment and communication costs.

Purpose of the Study:

• To introduce TRIP, a novel framework for FedDG that addresses limitations of current MoE-based prompt learning.
• To enable fine-grained visual pattern capture through token-level expert assignment.
• To reduce communication overhead via parameter-free routing.

Main Methods:

• TRIP employs a Token-level pRompt mIxture with Parameter-free routing framework.
• Individual image tokens are assigned to distinct prompt experts for specialized learning.
• Parameter-free routing utilizes capacity-aware clustering and Optimal Transport (OT) for efficient expert assignment.

Main Results:

• TRIP achieves optimal generalization performance across four benchmarks.
• The framework significantly reduces communication costs, requiring as few as 1K parameters.
• Token-level assignment captures fine-grained visual patterns more effectively than image-level methods.

Conclusions:

• TRIP offers an effective and efficient solution for Federated Domain Generalization.
• The proposed parameter-free routing mechanism drastically cuts communication overhead.
• TRIP demonstrates the potential of token-level expert assignment for specialized and generalizable models.