Frozen CLIP-DINO: A Strong Backbone for Weakly Supervised Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Weakly supervised semantic segmentation (WSSS) commonly uses image-level labels.
• Existing WSSS methods often generate pseudo-labels using models like CLIP for separate segmentation model training.
• No prior work directly employs CLIP as a backbone for direct object segmentation with image-level labels.

Purpose of the Study:

• To propose WeCLIP and WeCLIP+, single-stage pipelines for WSSS.
• To leverage frozen CLIP and DINO models for direct semantic feature extraction and segmentation.
• To introduce refinement modules for optimizing pseudo-labels and improving segmentation accuracy.

Main Methods:

• WeCLIP utilizes a frozen CLIP model as the backbone for feature extraction and a lightweight decoder for prediction.
• WeCLIP+ combines frozen CLIP and DINO models as a backbone for enhanced feature extraction.
• Both methods employ a refinement module (RFM/RFM+) to dynamically optimize pseudo-labels during training.

Main Results:

• WeCLIP and WeCLIP+ significantly outperform existing WSSS approaches.
• WeCLIP+ achieves high mean Intersection over Union (mIoU) scores: 83.9% on VOC 2012 and 56.3% on COCO.
• Both models demonstrate competitive performance in fully supervised settings.

Conclusions:

• WeCLIP and WeCLIP+ offer efficient and effective single-stage solutions for WSSS.
• The integration of CLIP and DINO models enhances feature representation for improved segmentation.
• These methods reduce training costs while achieving state-of-the-art performance.