SED++: A Simple Encoder-Decoder for Improved Open-Vocabulary Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Open-vocabulary semantic segmentation requires partitioning images into semantic regions using an open set of categories.
• Current methods often rely on image-level pre-trained vision-language models for pixel-level segmentation.
• There is a need for efficient and effective architectures for this task.

Purpose of the Study:

• To propose SED, a novel encoder-decoder architecture for open-vocabulary semantic segmentation.
• To leverage pre-trained vision-language models for improved segmentation performance.
• To enhance inference efficiency through architectural innovations.

Main Methods:

• SED employs a hierarchical image encoder and a text encoder to generate a cost volume.
• A gradual fusion decoder progressively integrates features and the cost volume for segmentation.
• A category early rejection strategy is integrated to filter irrelevant categories, boosting efficiency.

Main Results:

• SED achieves an mIoU of 34.9% on ADE20K (150 classes) with fast inference (69 ms/image).
• The method demonstrates strong performance on video semantic segmentation, reaching 40.2% mIoU on VSPW.
• Experiments validate the effectiveness and efficiency of the proposed SED architecture.

Conclusions:

• SED offers a simple yet effective solution for open-vocabulary semantic segmentation.
• The architecture's hierarchical design and early rejection strategy contribute to its performance and efficiency.
• SED shows promise for both image and video segmentation tasks, with potential for further extensions.