Fed-DiffLoRA: Personalized Federated Style Transfer for T2I Diffusion Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Centralized Low-Rank Adaptation (LoRA) merging for text-to-image (T2I) diffusion models poses privacy risks.
• Federated learning approaches for LoRA adaptations face challenges like model heterogeneity and vulnerability to inference attacks.

Purpose of the Study:

• To propose Fed-DiffLoRA, a privacy-preserving framework for secure aggregation of LoRA adapters in federated T2I diffusion models.
• To address the structural heterogeneity and member inference attack vulnerabilities inherent in diffusion models.

Main Methods:

• Disentangling client-specific LoRA adaptations into orthogonal content and style subspaces.
• Developing a learnable aggregation operator for dynamic, privacy-preserving fusion of style LoRAs based on content LoRA semantic vectors.
• Providing theoretical convergence and privacy guarantees.

Main Results:

• Fed-DiffLoRA significantly reduces attack success rates for member inference attacks.
• The framework achieves high-fidelity style blending while preserving semantic fidelity.
• Experimental validation confirms the effectiveness and privacy-preserving capabilities of the proposed approach.

Conclusions:

• Fed-DiffLoRA offers a robust solution for privacy-preserving federated customization of T2I diffusion models.
• The proposed method effectively balances stylization fidelity with enhanced privacy protection against sensitive attribute leakage.