CAPTR-GTP: Class-aware prompting and token refinement with graph token propagation for few-shot ViTs.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) excel at modeling long-range dependencies but face challenges in few-shot learning scenarios.
• Existing ViT adapters often discard or overcompress token information, leading to performance degradation, especially in cluttered scenes or with limited data.

Purpose of the Study:

• To introduce CAPTR-GTP, a novel framework designed to overcome the limitations of Vision Transformers in episodic few-shot learning.
• To preserve and effectively propagate token information, enhancing the model's ability to learn from scarce labeled data.

Main Methods:

• Utilized a pre-trained ViT encoder, largely frozen, for stable patch embeddings.
• Implemented an uncertainty-aware module with Monte Carlo dropout to estimate token reliability and a variance-consistent gate for noisy patches.
• Developed Class-Aware Prompt Refinement to align attention with class semantics and adapt to intra-class variations.
• Introduced a bi-level hierarchical attention module and Graph Token Propagation for refining tokens and aggregating context across clusters.

Main Results:

• CAPTR-GTP demonstrated state-of-the-art or consistently competitive performance across four in-domain and four cross-domain benchmarks.
• The framework achieved strong results under both 5-way 1-shot and 5-shot learning protocols.
• Preservation and propagation of token information proved effective in mitigating information loss and improving few-shot accuracy.

Conclusions:

• CAPTR-GTP offers a robust solution for few-shot learning with Vision Transformers by preserving and refining token information.
• The proposed methods effectively address issues like background noise amplification and reliance on static prototypes.
• The framework shows significant promise for advancing few-shot learning capabilities in computer vision.