Replay Master: Automatic Sample Selection and Effective Memory Utilization for Continual Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Continual Semantic Segmentation (CSS) trains models on incrementally introduced classes.
• Catastrophic forgetting is a key challenge in CSS, often addressed by replay methods using memory buffers.
• Existing replay methods struggle with optimal sample selection and effective utilization, and often ignore class imbalance.

Purpose of the Study:

• To develop a novel replay-based pipeline for Continual Semantic Segmentation (CSS).
• To address limitations in memory sample selection and utilization within existing replay methods.
• To mitigate the class imbalance problem inherent in limited memory replay strategies.

Main Methods:

• A reinforcement learning framework with novel state representations and a dual-stage action scheme for automatic memory sample selection.
• An expert mechanism and a dual-phase training method to manage class imbalance during replay.
• Integration of these components into a new replay-based pipeline for CSS.

Main Results:

• The proposed method achieves state-of-the-art (SOTA) performance on Pascal VOC 2012 and ADE20K datasets.
• Demonstrated significant improvements over previous advanced methods in Continual Semantic Segmentation.
• Validated the effectiveness of automatic sample selection and improved memory utilization.

Conclusions:

• The developed replay-based pipeline effectively enhances Continual Semantic Segmentation by optimizing memory sample selection and utilization.
• The novel approach successfully tackles catastrophic forgetting and class imbalance issues.
• This work presents a significant advancement in replay strategies for Continual Semantic Segmentation.