Static-dynamic class-level perception consistency in video semantic segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Video semantic segmentation (VSS) is crucial for applications like autonomous driving and surveillance.
• Leveraging temporal information remains a key challenge in VSS.
• Prior methods often focus on pixel-level context, limiting performance.

Purpose of the Study:

• To propose a novel framework for video semantic segmentation (VSS) that addresses the challenge of temporal information integration.
• To rethink static-dynamic contexts at the class level for improved segmentation.
• To introduce a computationally efficient attention mechanism for VSS.

Main Methods:

• Proposed a static-dynamic class-level perceptual consistency (SD-CPC) framework.
• Introduced multivariate class prototype with contrastive learning for class-level constraints.
• Developed a static-dynamic semantic alignment module using multi-scale and multi-level correlations.
• Implemented a window-based attention map calculation method to reduce computational cost.

Main Results:

• Achieved 51.1 mIoU on the VSPW dataset (MiT-B5).
• Attained 81.6 mIoU on Cityscapes and 78.2 mIoU on CamVid datasets (ResNet101).
• Outperformed existing state-of-the-art methods on benchmark datasets.

Conclusions:

• The proposed SD-CPC framework effectively leverages temporal information at the class level for superior video semantic segmentation.
• The novel methods for class-level context and semantic alignment significantly improve segmentation accuracy.
• The efficient attention mechanism contributes to practical VSS applications.