Prompt is All You Need: Prompting Foundation Models for Large-Scale Self-Supervised Semantic Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Large-scale unsupervised semantic segmentation (LUSS) is a challenging dense prediction task.
• Existing methods struggle with the scale and complexity of LUSS.

Purpose of the Study:

• To present simple, effective, and efficient solutions for LUSS using foundation models (FMs).
• To introduce Prompting foundation models for LUSS (PLUSS) as a novel approach.

Main Methods:

• Developed PLUSS_alpha, a cascade framework combining CLIP, Grounding DINO, and SAM in a zero-shot manner.
• Introduced PLUSS_beta with semantic and box tuner modules for enhanced prompt quality, using self-supervised signals from FMs.
• No external supervision or FM parameter updates were required.

Main Results:

• PLUSS_alpha established a strong baseline, outperforming prior state-of-the-art methods.
• PLUSS_beta achieved significant improvements: 39.6% (50 categories), 27.3% (300 categories), and 22.6% (919 categories) in mIoU on ImageNet-S.
• Demonstrated robust category-shape representation and strong generalization for open-vocabulary tasks.

Conclusions:

• PLUSS provides a powerful and efficient framework for adapting foundation models to downstream vision tasks.
• The proposed method sets a new benchmark for large-scale unsupervised semantic segmentation.