RefSAM: Efficiently adapting segmenting anything model for referring video object segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• The Segment Anything Model (SAM) demonstrates strong image segmentation capabilities but struggles with referring video object segmentation (RVOS).
• Existing RVOS methods often require precise user prompts and lack robust multi-modal understanding (language and vision).

Purpose of the Study:

• To adapt SAM for effective referring video object segmentation (RVOS).
• To enhance cross-modality learning by integrating diverse visual and linguistic information from successive video frames.

Main Methods:

• Introduced the RefSAM model, adapting SAM with a Cross-Modal MLP for text-to-embedding projection.
• Developed a hierarchical dense attention module for fusing visual-semantic information and sparse embeddings.
• Incorporated an implicit tracking module for historical context and a parameter-efficient tuning strategy for feature alignment.

Main Results:

• RefSAM effectively incorporates multi-view information from diverse modalities and successive frames.
• The model demonstrates superior performance on RVOS tasks compared to existing methods.
• Ablation studies confirm the efficacy of the proposed design choices.

Conclusions:

• RefSAM significantly advances the application of SAM for referring video object segmentation.
• The model's ability to fuse multi-modal information and leverage temporal context offers a robust solution for RVOS.