Boosting Semantic Segmentation by Conditioning the Backbone with Semantic Boundaries

Area of Science:

• Computer Vision
• Deep Learning

Background:

• Semantic segmentation models often struggle with precise mask delineation, particularly at object boundaries.
• Existing methods may require complex post-processing or introduce significant computational overhead.

Purpose of the Study:

• To introduce a novel framework, the Semantic-Boundary-Conditioned Backbone (SBCB), to enhance semantic segmentation performance.
• To specifically improve the accuracy of segmentation masks around object boundaries.
• To ensure compatibility with diverse segmentation architectures and avoid inference-time complexity.

Main Methods:

• Proposed the Semantic-Boundary-Conditioned Backbone (SBCB) framework.
• Integrated a complementary semantic boundary detection (SBD) task using a multi-task learning approach.
• Utilized multi-scale features within the SBD head to capture both low-level and high-level semantic information.
• Ensured the framework enhances the backbone without additional inference parameters or post-processing.

Main Results:

• Achieved an average improvement of 1.2% in Intersection over Union (IoU) on the Cityscapes dataset.
• Demonstrated a 2.6% gain in boundary F-score, indicating improved boundary localization.
• Showcased enhanced performance in addressing over- and under-segmentation issues.
• Validated effectiveness across various segmentation heads, backbones, and emerging vision transformer models.

Conclusions:

• The SBCB framework effectively boosts semantic segmentation performance, especially at mask boundaries.
• The auxiliary SBD task improves segmentation accuracy without increasing model complexity or inference cost.
• The SBCB framework shows broad applicability and consistent performance gains across different architectures and benchmarks.