TagCLIP: Improving Discrimination Ability of Zero-Shot Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Contrastive Language-Image Pre-training (CLIP) shows potential in pixel-level zero-shot learning.
• Current methods struggle with distinguishing unseen classes, confusing them with similar ones.

Purpose of the Study:

• To propose TagCLIP, a novel approach to improve pixel-level zero-shot learning.
• To enhance the discrimination ability of CLIP in identifying novel classes.

Main Methods:

• Disentangling the optimization problem into semantic matching and reliability judgment.
• Introducing a 'trusty token' for improved class distinction, inspired by language modeling.
• Evaluating TagCLIP on PASCAL VOC 2012 and COCO-Stuff 164 K datasets.

Main Results:

• TagCLIP significantly improves Intersection over Union (IoU) for unseen classes.
• Achieved 7.4% and 1.7% IoU improvement on benchmark datasets.
• Demonstrated negligible computational overheads.

Conclusions:

• TagCLIP effectively addresses the confusion between novel and similar classes in zero-shot learning.
• The trusty token mechanism enhances the reliability of semantic mask generation.
• TagCLIP offers a promising solution for more accurate pixel-level zero-shot tasks.