Enhancing Few-Shot CLIP With Semantic-Aware Fine-Tuning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks struggle with generalized representation learning from limited data in low-resource scenarios.
• Contrastive language-image pretraining (CLIP) shows promise in few-shot adaptation, but freezing its parameters can hinder performance on downstream tasks.
• Existing methods often freeze CLIP parameters to prevent overfitting and catastrophic forgetting, potentially ignoring task-specific semantic needs.

Purpose of the Study:

• To improve few-shot learning performance by adapting CLIP's visual encoder to downstream tasks.
• To address the limitation of fixed attention pooling in CLIP for diverse few-shot scenarios.
• To develop a method that fine-tunes task-specific semantics while retaining CLIP's general knowledge.

Main Methods:

• Proposed fine-tuning the attention pooling layer of CLIP's visual encoder to focus on task-specific semantics.
• Introduced residual blending during inference to combine fine-tuned and original CLIP features.
• Developed semantic-aware fine-tuning (SAF) and integrated it with adapter methods (termed SAF-Adapter).

Main Results:

• Semantic-aware fine-tuning (SAF) significantly enhances few-shot CLIP performance across 11 benchmarks.
• SAF and SAF-Adapter outperform the second-best methods by substantial margins in one-shot (1.51-2.38%) and four-shot (0.48-1.37%) settings.
• The proposed methods demonstrate effectiveness in low-resource few-shot adaptation tasks.

Conclusions:

• Fine-tuning CLIP's attention pooling layer is a viable strategy for improving few-shot learning.
• Semantic-aware fine-tuning effectively adapts models to task-specific nuances, boosting generalization.
• The proposed methods offer significant improvements for few-shot adaptation in deep learning.