Multi-Granularity Context Network for Efficient Video Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video semantic segmentation faces challenges in utilizing multi-frame context and maintaining computational efficiency.
• Existing methods often struggle to balance context aggregation with processing speed.

Purpose of the Study:

• To develop a novel network, the Multi-Granularity Context Network (MGCNet), that addresses both context utilization and computational efficiency in video semantic segmentation.
• To enhance feature representation by aggregating semantic information at multiple granularities.

Main Methods:

• The MGCNet converts image features into semantic prototypes and uses a non-local operation to jointly aggregate per-frame and short-term contexts.
• A long-term context module captures video-level semantics during training.
• An uncertainty-aware and structural knowledge distillation strategy is employed to further improve performance.

Main Results:

• The proposed pixel-to-prototype non-local operation is computationally less expensive than traditional methods and reuses semantic prototypes for efficiency.
• MGCNet achieves state-of-the-art performance on Cityscapes and CamVid datasets.
• The method demonstrates high speed and low latency.

Conclusions:

• MGCNet effectively integrates multi-granularity context for robust video semantic segmentation.
• The network offers a significant improvement in both performance and computational efficiency compared to existing methods.