Rethinking Self-Supervised Semantic Segmentation: Achieving End-to-End Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Semantic segmentation typically requires extensive pixel-level annotations, posing a significant challenge.
• Existing self-supervised methods often train image encoders or segmentation heads but rely on supervised classifiers or clustering for inference, hindering real-time application.
• Dataset-level clustering for segmentation is inefficient and degrades performance by treating all pixels collectively.

Purpose of the Study:

• To propose a novel self-supervised semantic segmentation paradigm enabling end-to-end training and inference.
• To address semantic inconsistencies and poor representation quality in non-salient regions observed in self-supervised Vision Transformers (ViT).
• To develop a method that performs segmentation inference adaptively per image.

Main Methods:

• Proposing prototype-image alignment and global-local alignment with an attention map constraint.
• Training a Transformer Decoder with learnable prototypes.
• Utilizing adaptive prototypes for per-image segmentation inference.

Main Results:

• Demonstrated superior performance in fully unsupervised semantic segmentation settings.
• Showcased the generalizability of the proposed method across different datasets.
• Achieved end-to-end inference, overcoming limitations of prior clustering-based approaches.

Conclusions:

• The proposed self-supervised semantic segmentation method offers a viable solution for scenarios with limited annotations.
• The novel alignment strategies and adaptive prototypes significantly enhance segmentation accuracy and efficiency.
• The approach facilitates real-time, end-to-end inference, advancing the field of unsupervised semantic segmentation.