Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Custom diffusion models (CDMs) excel at personalized concept generation but struggle with static concept sets and incremental learning.
• Existing CDMs often suffer from catastrophic forgetting and concept neglect when adapting to new information.
• The need for dynamic and adaptable personalized concept generation in diffusion models is critical.

Purpose of the Study:

• To develop a novel Continually Customizable Diffusion Model (CCDM) that supports concept-incremental versatile customization.
• To mitigate catastrophic forgetting and concept neglect in the context of continual learning for CDMs.
• To enhance the consistency and semantic independence of multi-concept synthesis in personalized generation.

Main Methods:

• Introduced an attribute-decoupled LoRA (AD-LoRA) module to preserve concept-specific attributes and leverage inter-task correlations.
• Developed a relevance-guided AD-LoRA aggregation strategy to combat catastrophic forgetting.
• Proposed a controllable regional context synthesis strategy for improved multi-concept composition and consistency.

Main Results:

• The proposed CCDM demonstrated significant improvements over baseline methods in continual learning scenarios.
• The AD-LoRA module and aggregation strategy effectively preserved previously learned concepts.
• The regional context synthesis strategy enhanced multi-concept composition and consistency, reducing concept neglect.

Conclusions:

• The CCDM offers a robust solution for dynamic and incremental personalization in diffusion models.
• The developed methods effectively address catastrophic forgetting and concept neglect in continual learning.
• CCDM enables versatile and consistent multi-concept synthesis for personalized content generation.