Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Pre-trained vision-language models like CLIP enable open-vocabulary segmentation with strong zero-shot capabilities.
• Vision Transformer (ViT)-based CLIP models struggle with local details in segmentation due to global feature interactions and anomaly tokens.
• Anomaly tokens in ViT-based CLIP diminish spatial awareness and hinder segmentation performance.

Purpose of the Study:

• To propose a training-free method, Self-Calibrated CLIP (SC-CLIP), to improve the performance of CLIP for segmentation tasks.
• To address the issue of anomaly tokens and enhance the capture of local details in vision-language models.
• To calibrate CLIP for finer representations without introducing new parameters or additional backbones.

Main Methods:

• SC-CLIP identifies and replaces anomaly tokens based on local context.
• It reduces the influence of anomaly tokens by enhancing feature discriminability and attention correlation.
• A two-pass strategy integrates multi-level features to enrich local details in a training-free manner.

Main Results:

• SC-CLIP achieves state-of-the-art results across all tested datasets, surpassing previous methods by 9.5%.
• The method significantly boosts the performance of vanilla CLIP ViT-L/14 by 6.8 times.
• SC-CLIP enhances CLIP's feature representations with improved granularity and semantic coherence.

Conclusions:

• SC-CLIP effectively calibrates CLIP for improved segmentation performance by addressing anomaly tokens and enhancing local details.
• The training-free approach preserves CLIP's generalization ability while significantly boosting performance.
• SC-CLIP demonstrates broad applicability to other vision-language models and tasks.