MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Open-world semantic segmentation requires extensive, precisely annotated image-text datasets.
• Current data acquisition is costly and time-consuming, limiting model training.
• Diffusion models offer powerful image generation capabilities.

Purpose of the Study:

• To introduce MagicSeg, a novel pipeline for automated dataset generation for open-world semantic segmentation.
• To leverage diffusion models for creating high-fidelity images and counterfactual negative samples.
• To enhance open-world semantic segmentation performance by addressing data scarcity and annotation challenges.

Main Methods:

• MagicSeg generates textual descriptions from class labels to guide diffusion models in image creation.
• It produces both positive and negative (counterfactual) image samples for contrastive training.
• Integration with open-vocabulary detection and interactive segmentation models extracts pseudo-masks for self-supervised pretraining.

Main Results:

• The generated dataset, when applied to contrastive language-image pretraining, significantly boosts downstream model performance.
• State-of-the-art (SOTA) results were achieved on PASCAL VOC (62.9%), PASCAL Context (26.7%), and COCO (40.2%).
• Demonstrates the effectiveness of the MagicSeg dataset in improving open-world semantic segmentation capabilities.

Conclusions:

• MagicSeg offers an efficient, automated solution for creating valuable datasets for open-world semantic segmentation.
• The proposed method effectively overcomes the limitations of traditional data annotation.
• The generated dataset enables significant advancements in open-world semantic segmentation performance.