UAMP: Consistent video object segmentation with uncertainty-aware memory propagation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Segment Anything Model 2 (SAM 2) is a foundational model for video object segmentation.
• SAM 2 struggles with crowded scenes, fast-moving objects, self-occlusion, and error accumulation in its memory architecture.
• These limitations hinder consistent object segmentation in complex video scenarios.

Purpose of the Study:

• To enhance the consistency and robustness of SAM 2 for video object segmentation.
• To address limitations in crowded scenarios, object occlusion, and error accumulation.
• To introduce the UAMP method for improved video object segmentation.

Main Methods:

• Proposed UAMP, an enhanced SAM 2 variant.
• Integrated memory propagation with explicit appearance and motion uncertainty modeling.
• Implemented a dual mechanism for long-term memory updating and short-term memory selection.
• Fused uncertainty-aware representations with dual memory mechanisms.

Main Results:

• UAMP demonstrated superior performance in quantitative and qualitative evaluations on benchmark datasets.
• Achieved significant improvements in scenarios with occlusions and object reappearances.
• Showcased consistent performance gains over state-of-the-art methods across five VOS benchmarks, with up to a 5.6-point J&F metric enhancement.

Conclusions:

• UAMP effectively accommodates dynamic object appearance and motion variations.
• The method refines the object memory bank, realizing consistent video object segmentation.
• UAMP provides a robust and effective solution for complex tracking scenarios, enhancing SAM 2 for practical applications.