Fed-DiffLoRA: Personalized Federated Style Transfer for T2I Diffusion Models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Low-Rank Adaptation (LoRA) merging is efficient for customizing text-to-image (T2I) diffusion models.
• Centralized aggregation of LoRA adapters poses significant privacy risks.
• Federated learning approaches for LoRA adaptations face challenges like structural heterogeneity and vulnerability to member inference attacks in diffusion models.

Purpose of the Study:

• To propose Fed-DiffLoRA, a novel privacy-preserving framework for securely aggregating LoRA adapters in federated T2I diffusion models.
• To address the unique challenges of structural heterogeneity and privacy vulnerabilities in federated diffusion models.
• To enable high-fidelity style blending while mitigating privacy risks associated with federated learning.

Main Methods:

• Disentangling client-specific LoRA adaptations into orthogonal content LoRAs (semantic fidelity) and style LoRAs (stylistic features).
• Designing a learnable aggregation operator for dynamic, cross-client style LoRA fusion based on content LoRA semantic vectors.
• Providing theoretical guarantees for convergence and privacy preservation.

Main Results:

• Achieved high-fidelity style blending by dynamically optimizing style LoRA fusion.
• Demonstrated substantial reductions in member inference attack success rates, enhancing privacy.
• Maintained consistent and high stylization fidelity across experiments.

Conclusions:

• Fed-DiffLoRA effectively enhances privacy in federated T2I diffusion model customization.
• The proposed method successfully balances semantic fidelity and stylistic accuracy with robust privacy protection.
• The framework offers a viable solution for secure and efficient collaborative customization of diffusion models.