Learning Local and Global Temporal Contexts for Video Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Contextual information is crucial for accurate video semantic segmentation (VSS).
• Existing methods often focus on either local temporal contexts (LTC) or global temporal contexts (GTC), but rarely both.
• Simultaneously learning static and motional contexts within LTC offers complementary benefits.

Purpose of the Study:

• To propose a unified approach for learning local temporal contexts (LTC) in VSS.
• To introduce a method for incorporating global temporal contexts (GTC) to further improve VSS performance.
• To enhance the accuracy and effectiveness of video semantic segmentation.

Main Methods:

• Coarse-to-Fine Feature Mining (CFFM) technique to learn a unified representation of LTC.
• CFFM comprises Coarse-to-Fine Feature Assembling (CFFA) for abstracting static and motional contexts, and Cross-frame Feature Mining (CFM) for enhancing features from neighboring frames.
• CFFM++ extends CFFM by incorporating GTC through k-means clustering of sampled frames and CFM for prototype refinement.

Main Results:

• The proposed CFFM method effectively learns a unified presentation of local temporal contexts.
• CFFM++ demonstrates improved performance by additionally leveraging global temporal contexts.
• Both CFFM and CFFM++ achieve state-of-the-art results on popular video semantic segmentation benchmarks.

Conclusions:

• Simultaneously learning static and motional contexts via CFFM significantly enhances VSS.
• Integrating global temporal contexts with CFFM++ further boosts segmentation accuracy.
• The proposed methods offer a more comprehensive approach to exploiting temporal information for VSS.