Structure-Induced Gradient Regulation for Generalizable Vision-Language Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Prompt tuning adapts pre-trained vision-language models efficiently using soft prompts.
• Few-shot learning with prompt tuning faces challenges in initialization sensitivity and overfitting.
• Existing methods struggle with rapid adaptation and maintaining generalizability in low-data scenarios.

Purpose of the Study:

• Introduce a novel Gradient-RegulAted Meta-prompt learning (GRAM) framework.
• Enhance prompt tuning's effectiveness in few-shot and zero-shot learning scenarios.
• Improve cross-domain generalizability and reduce overfitting in vision-language models.

Main Methods:

• Developed a meta-learning paradigm using weakly labeled image-text data.
• Employed Cross-Modal Hierarchical Clustering for data organization.
• Introduced a gradient regulating function for improved generalizability.
• Proposed a structure-induced gradient regulating function for test-time tuning.

Main Results:

• GRAM achieves state-of-the-art few-shot and zero-shot generalizability.
• The framework consistently improves various prompt tuning methods.
• Demonstrated effective adaptation with limited or unlabeled data.
• Showcased robust meta-learning across diverse domains.

Conclusions:

• GRAM offers a robust and adaptable solution for prompt tuning challenges.
• The proposed methods significantly advance few-shot and zero-shot learning capabilities.
• GRAM provides a model-agnostic approach for enhancing prompt tuning performance.
• The framework facilitates efficient knowledge transfer without explicit annotations.