CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Visual-text pretrained models like CLIP excel in vision tasks.
• CLIP's capabilities are being explored for pixel-level tasks, including few-shot segmentation.
• Existing few-shot segmentation methods rely on support and query features for class prototypes.

Purpose of the Study:

• To explore CLIP's potential in few-shot segmentation.
• To propose a new method leveraging CLIP's text features for enhanced semantic information.
• To improve prototype generation for better feature matching and segmentation accuracy.

Main Methods:

• Utilize CLIP to extract text features for specific classes, incorporating them into the training process.
• Propose a novel prototype generation method using multi-modal fusion of text and image features.
• Generate adaptive query prototypes by combining image foreground/background information with multi-modal support prototypes.

Main Results:

• The proposed method effectively extracts richer semantic information using text features.
• Multi-modal fusion in prototype generation improves matching of query image features.
• Achieved excellent results on PASCAL-5i and COCO-20i few-shot segmentation datasets.

Conclusions:

• The study offers a new perspective on few-shot segmentation in multi-modal scenarios.
• Integrating CLIP's text features significantly enhances segmentation performance.
• The proposed method demonstrates state-of-the-art results, validating its effectiveness.