Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Large vision-language models (e.g., CLIP) excel at zero-shot transfer.
• Few-shot recognition requires adapting models to limited data.
• Existing methods like Tip-Adapter improve few-shot performance but face scalability issues due to large adapter sizes.

Purpose of the Study:

• To propose a novel, efficient adaptation method for CLIP.
• To develop a constant-size adapter that overcomes Tip-Adapter's scalability limitations.
• To enhance few-shot classification performance with a focus on discriminative decision boundaries.

Main Methods:

• Introduced Proto-Adapter, a single-layer adapter with constant size.
• Constructed adapter weights using prototype representations derived from class feature aggregation.
• Implemented a distance margin penalty during fine-tuning to increase inter-class discrepancy.

Main Results:

• Proto-Adapter achieved superior few-shot recognition performance compared to Tip-Adapter.
• The proposed method demonstrated effectiveness across diverse datasets in few-shot classification tasks.
• The constant adapter size ensures efficient deployment regardless of training data volume.

Conclusions:

• Proto-Adapter offers an effective and scalable solution for few-shot adaptation of large vision-language models.
• The prototype-based approach and distance margin penalty contribute to improved model discriminability.
• This method facilitates practical applications requiring efficient few-shot learning with limited data.