Toward Generative Understanding: Incremental Few-Shot Semantic Segmentation With Diffusion Models

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Incremental Few-shot Semantic Segmentation (iFSS) addresses continual learning challenges, including catastrophic forgetting of base classes when learning new ones.
• Existing iFSS methods struggle with feature drift and limited generalization due to bidirectional coupling bottlenecks.
• Current techniques like knowledge distillation and background learning show partial effectiveness but require improvement.

Purpose of the Study:

• To propose a novel diffusion-based generative framework for iFSS.
• To overcome limitations of existing methods, specifically feature drift and catastrophic forgetting.
• To enable efficient and effective learning of novel classes with limited data while preserving base class knowledge.

Main Methods:

• A diffusion-based generative framework is introduced, bridging generative and discriminative tasks.
• An innovative binary-to-RGB mask mapping mechanism utilizes pre-trained diffusion models.
• Class-specific semantic embedding optimization and color embeddings focus on target regions and enhance contrast, followed by a lightweight post-processor for mask refinement.

Main Results:

• The proposed framework achieves state-of-the-art performance on PASCAL-5ⁱ and COCO-20ⁱ datasets.
• It requires significantly less data compared to existing methods.
• Demonstrates competitive results in cross-domain few-shot segmentation tasks, showcasing strong generalization capabilities.

Conclusions:

• The diffusion-based framework effectively addresses catastrophic forgetting and feature drift in iFSS.
• Leveraging diffusion priors and optimized embeddings enables rapid novel-class adaptation and robust performance.
• This approach offers a promising direction for continual learning in semantic segmentation.