iSeg: An Iterative Refinement-based Framework for Training-free Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Stable Diffusion models excel at text-to-image synthesis, indicating inherent semantic understanding.
• Existing training-free segmentation methods partially utilize self-attention maps for refinement.
• There's a need for methods that fully leverage self-attention map information for improved segmentation.

Purpose of the Study:

• To present an effective iterative refinement framework for training-free segmentation, named iSeg.
• To fully utilize self-attention map information for enhancing segmentation performance.
• To introduce novel modules for improved attention map generation and refinement.

Main Methods:

• Proposing iSeg, an iterative refinement framework for training-free segmentation.
• Introducing an entropy-reduced self-attention module using gradient descent to suppress irrelevant global information.
• Designing a category-enhanced cross-attention module for accurate initial attention map generation.

Main Results:

• iSeg stably improves cross-attention maps through iterative refinement.
• Achieved an absolute gain of 3.8% in mIoU for unsupervised semantic segmentation on Cityscapes compared to prior work.
• Demonstrated promising performance across diverse segmentation tasks, including weakly-supervised, open-vocabulary, and unsupervised segmentation.

Conclusions:

• The proposed iSeg framework effectively enhances training-free segmentation by iteratively refining attention maps.
• The entropy-reduced self-attention and category-enhanced cross-attention modules are key contributions to improved performance.
• iSeg offers flexibility, supporting various segmentation tasks and integration into different frameworks.