Robust Adaptation of Foundation Models With Black-Box Visual Prompting

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large-scale pre-trained models (PTMs) dominate AI, but parameter-efficient transfer learning (PETL) faces challenges.
• Real-world applications often lack full PTM parameter access (black-box APIs) and have memory constraints.
• Existing PETL methods assume full model access and substantial memory for gradient computation.

Purpose of the Study:

• To propose a novel method, BlackVIP, for efficient adaptation of PTMs as black-box APIs.
• To enable parameter-efficient transfer learning without requiring full model parameter knowledge or high memory usage.
• To address the limitations of current PETL approaches in practical, resource-constrained scenarios.

Main Methods:

• BlackVIP utilizes input-dependent visual prompts generated by a Coordinator module.
• Simultaneous Perturbation Stochastic Approximation with Gradient Correction (SPSA-GC) is employed for efficient gradient estimation.
• A variant, BlackVIP-SE, is introduced to further reduce computational cost and runtime.

Main Results:

• BlackVIP demonstrated robust adaptation across 19 diverse datasets and tasks.
• The method achieves effective transfer learning with minimal memory requirements.
• Theoretical analysis connects visual prompting to certified robustness, supported by empirical evidence of improved robustness.

Conclusions:

• BlackVIP offers an efficient and memory-light solution for adapting large pre-trained models as black-box APIs.
• The approach overcomes key limitations of traditional PETL methods in real-world settings.
• BlackVIP shows promise for enhancing model robustness and generalization in diverse applications.