From Pixels to Semantics: Self-Supervised Video Object Segmentation With Multiperspective Feature Mining.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Current self-supervised methods for one-shot video object segmentation (O-VOS) frame the task as pixel-level matching.
• This approach is limited because O-VOS requires semantic correspondence more than precise pixel matching.

Purpose of the Study:

• To develop a novel self-supervised framework that integrates pixel-level correspondence learning with semantic-level adaptation for improved O-VOS.
• To enhance feature reliability and suppress noise for more robust image matching in video segmentation.

Main Methods:

• Implemented a self-supervised framework combining pixel-level correspondence learning via photometric reconstruction of adjacent RGB frames during offline training.
• Incorporated semantic-level adaptation at test-time by enforcing bi-directional agreement of predicted segmentation masks.
• Proposed a new network architecture featuring a multi-perspective feature mining mechanism to enhance reliable features and reduce noisy ones.

Main Results:

• Achieved state-of-the-art performance on widely adopted datasets for one-shot video object segmentation.
• Demonstrated the effectiveness of the integrated pixel-level and semantic-level adaptation approach.
• Showcased the benefits of the multi-perspective feature mining mechanism for robust image matching.

Conclusions:

• The proposed self-supervised framework effectively bridges the gap between self-supervised and fully supervised methods in O-VOS.
• The integration of semantic-level adaptation significantly improves segmentation mask propagation.
• The novel network architecture contributes to more robust and accurate video object segmentation.