Rethinking Feature Reconstruction via Category Prototype in Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Encoder-decoder architectures are standard for semantic segmentation.
• Aggregating multi-stage encoder features is crucial for discriminative pixel representation.
• Existing methods often rely on limited receptive fields for feature reconstruction.

Purpose of the Study:

• To propose a novel Query Update (Q-UP) task for scale alignment of multi-stage pyramidal features.
• To introduce Category Prototype Transformer (CPT) for improved semantic segmentation.
• To reduce computational complexity while maintaining high performance.

Main Methods:

• Q-UP dynamically broadcasts low-resolution features to higher resolutions using pixel-wise affinity scores and global receptive fields.
• Category prototypes, averaged from pixel features within a category, replace source pixel features for reconstruction.
• A memory module is utilized to explore dataset-level category prototype capacity.

Main Results:

• CPT integrated into a feature pyramid structure shows superior semantic segmentation performance, even with 1/32 resolution feature maps.
• The method achieves 55.5% mIoU on the ADE20K dataset.
• CPT significantly reduces model parameters and computational complexity compared to prior works.

Conclusions:

• Category Prototype Transformer (CPT) offers an effective approach for semantic segmentation by enhancing feature reconstruction.
• The proposed Q-UP task and category prototypes enable efficient scale alignment and reduce intra-category variance.
• CPT demonstrates a promising direction for developing computationally efficient and high-performance semantic segmentation models.