MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Context aggregation is crucial for semantic segmentation, but existing methods primarily focus on intra-image contexts.
• Valuable semantic information from categories outside the current image is often overlooked.

Purpose of the Study:

• To propose MCIBI++ (Mining Contextual Information Beyond Image), a novel paradigm to enhance pixel-level representations for semantic segmentation.
• To incorporate dataset-level category semantics into the segmentation process.

Main Methods:

• A dynamically updated memory module stores dataset-level category distribution information.
• Dataset-level category representations are generated and used as weights for context aggregation.
• Pixel representations are augmented with both dataset-level and image-level contextual information.
• A coarse-to-fine iterative inference strategy is employed during inference.

Main Results:

• MCIBI++ consistently improves performance when integrated into existing semantic segmentation frameworks.
• The method shows considerable improvements when extended to video semantic segmentation.
• State-of-the-art performance was achieved on seven challenging image and video semantic segmentation benchmarks.

Conclusions:

• MCIBI++ effectively leverages dataset-level contextual information to boost semantic segmentation performance.
• The proposed method offers a flexible and powerful extension for both image and video semantic segmentation tasks.
• MCIBI++ represents a significant advancement in improving pixel-level representations through cross-image semantic understanding.