DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• In-context segmentation, a subset of few-shot learning, requires segmenting objects from a single labeled example.
• Existing Segment Anything Models (SAMs) excel at interactive segmentation but are not directly suited for in-context segmentation tasks.
• Adapting SAMs for in-context segmentation is crucial for advancing generalization capabilities in computer vision.

Purpose of the Study:

• To propose Dual Consistency SAM (DC-SAM), a novel method for in-context segmentation in images and videos.
• To enhance the prompt encoder's feature representation within SAMs using high-quality visual prompts.
• To introduce the first benchmark for in-context video object segmentation (IC-VOS).

Main Methods:

• DC-SAM employs prompt-tuning to adapt SAM and SAM2 for in-context segmentation.
• Key techniques include fusing SAM features for better prompt alignment and cycle-consistent cross-attention for feature-visual prompt consistency.
• A dual-branch design with discriminative prompts and a mask-tube training strategy are utilized.

Main Results:

• DC-SAM achieves 55.5 mIoU on COCO-2i and 73.0 mIoU on PASCAL-5i datasets.
• The method attains a J&F score of 71.52 on the newly curated IC-VOS benchmark.
• Experiments validate the effectiveness of DC-SAM in both image and video in-context segmentation.

Conclusions:

• DC-SAM successfully adapts SAMs for effective in-context segmentation in images and videos.
• The proposed method demonstrates superior performance on standard few-shot segmentation benchmarks and the novel IC-VOS dataset.
• DC-SAM offers a significant advancement in the field of few-shot learning and visual segmentation.