JointFormer: A Unified Framework With Joint Modeling for Video Object Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current video object segmentation (VOS) methods use a decoupled extraction-then-matching pipeline.
• This approach limits frame-to-frame information propagation to high-level features, hindering fine-grained detail capture and robustness against distractors.

Purpose of the Study:

• To propose a unified Video Object Segmentation (VOS) framework, JointFormer, that jointly models feature extraction, correspondence matching, and memory.
• To enhance VOS performance by enabling extensive multi-layer feature propagation and incorporating long-term, holistic target information.

Main Methods:

• JointFormer employs a Joint Modeling Block using attention operations for simultaneous feature extraction and propagation.
• A compressed memory token with an online updating mechanism aggregates target features for frame-wise temporal information propagation.
• The framework facilitates instance-distinctive feature learning and global modeling consistency.

Main Results:

• JointFormer achieved new state-of-the-art performance on DAVIS 2017 and YouTube-VOS benchmarks.
• The model demonstrated excellent generalization and robustness, achieving top performance on diverse new benchmarks (MOSE, VISOR, VOST, LVOS).
• Ablation studies confirmed JointFormer's effectiveness in comprehensive feature learning and matching.

Conclusions:

• The unified JointFormer framework significantly advances Video Object Segmentation capabilities.
• Its joint modeling approach and compressed memory mechanism offer superior performance and generalization across various VOS challenges.
• JointFormer provides a more effective and robust solution for complex video object segmentation tasks.