Adapting Vision-Language Models via Learning to Inject Knowledge

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Pre-trained vision-language models (VLMs) like CLIP excel at zero-shot tasks due to extensive image-text training.
• Current methods often rely on task-specific prompts, limiting VLM adaptability to unseen tasks.
• VLMs possess significant memorized appearance knowledge within their text encoders.

Purpose of the Study:

• To develop a generalizable adaptation framework for VLMs to downstream vision tasks.
• To overcome limitations of task-specific prompts by leveraging task-agnostic knowledge.
• To improve VLM performance in few-shot learning, generalization, and domain transfer.

Main Methods:

• Proposes a knowledge injection framework using task-agnostic knowledge features.
• Extracts multi-layer features from learnable prompt sentences via VLM's text encoder.
• Introduces a knowledge injection module (KIM) to refine image or text features with extracted knowledge features.

Main Results:

• The proposed framework significantly enhances discriminative capability and robustness to intra-category variances.
• Achieved superior performance compared to recent methods in few-shot learning, base-to-new classes generalization, cross-dataset transfer, and domain generalization.
• Outperformed CoOp by 4.5% in few-shot learning and CoCoOp by 4.4% in base-to-new classes generalization.

Conclusions:

• The knowledge injection framework enables both modalities to benefit from the VLM's text encoder knowledge.
• Demonstrates effective and generalizable adaptation of VLMs to diverse downstream vision tasks.
• The approach offers a promising direction for improving VLM performance and adaptability.