Prompt-and-Transfer: Dynamic Class-Aware Enhancement for Few-Shot Segmentation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Few-shot Segmentation (FSS) models typically use fixed pre-trained encoders, leading to class-agnostic feature activation and irrelevant object detection.
• Human visual perception effectively focuses on specific objects, a capability lacking in current FSS approaches.
• The need for more adaptable and class-aware feature extraction in FSS is critical for improved generalization.

Purpose of the Study:

• To propose a novel prompt-driven scheme, Prompt and Transfer (PAT), mimicking human visual perception for Few-shot Segmentation.
• To develop a dynamic class-aware prompting paradigm to tune feature encoders for precise object focus.
• To enhance FSS performance across standard, cross-domain, weak-label, and zero-shot segmentation tasks.

Main Methods:

• Introduced a prompt-driven scheme (PAT) to dynamically tune feature encoders for class-specific focus.
• Incorporated cross-modal linguistic information for prompt initialization.
• Utilized Semantic Prompt Transfer (SPT) to transfer image-specific semantics to prompts and a Part Mask Generator (PMG) for adaptive part prompt generation.

Main Results:

• Achieved competitive performance on four distinct FSS tasks: standard FSS, Cross-domain FSS, Weak-label FSS, and Zero-shot Segmentation.
• Established new state-of-the-art results on 11 benchmarks across these diverse FSS domains.
• Demonstrated the efficacy of the PAT scheme in improving generalization to unseen classes and domains.

Conclusions:

• The proposed Prompt and Transfer (PAT) scheme offers a powerful and effective approach to Few-shot Segmentation.
• PAT's dynamic class-aware prompting paradigm significantly enhances feature encoder adaptability and object focus.
• The method demonstrates broad applicability and superior performance across various challenging segmentation scenarios.