Prototype-Based Semantic Segmentation.

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• Deep learning has significantly advanced semantic segmentation, utilizing diverse architectures and decoding schemes.
• Existing methods often interpret class representations as learnable prototypes, leading to inherent limitations.

Purpose of the Study:

• To address limitations in parametric semantic segmentation by proposing a nonparametric alternative.
• To develop a novel framework that utilizes non-learnable prototypes for improved pixel-wise prediction.

Main Methods:

• A nonparametric approach representing each class with non-learnable prototypes derived from training pixel features.
• Pixel-wise prediction achieved through nonparametric nearest prototype retrieval.
• Optimization of the pixel embedding space by aligning embedded pixels with anchored prototypes.

Main Results:

• The nonparametric framework demonstrates superior performance on standard datasets like ADE20K, Cityscapes, and COCO-Stuff.
• Effective application across various semantic segmentation models (FCN, Transformer-based) and backbones.
• Successful implementation in large-vocabulary semantic segmentation scenarios.

Conclusions:

• The proposed nonparametric method offers a scalable and effective alternative to parametric approaches in semantic segmentation.
• This framework accommodates an arbitrary number of classes with a constant number of learnable parameters.
• The study encourages a re-evaluation of current semantic segmentation model designs.