Enhancing Zero-Shot Adversarial Robustness of Vision-Language Models With Training-Free Adaptive Feature Movement.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Pre-trained Vision-Language Models (VLMs) excel at zero-shot tasks but are vulnerable to adversarial attacks.
• Current defense methods often require dataset-specific fine-tuning, compromising true zero-shot capabilities and risking overfitting.

Purpose of the Study:

• To develop a truly zero-shot and training-free method for improving the adversarial robustness of VLMs.
• To enhance VLM resilience against adversarial samples without compromising generalization abilities.

Main Methods:

• Introduced a method involving adding Gaussian noise to adversarial samples, using them as anchors.
• Developed a technique to find a path in the embedding space from noisy anchors to cleaner samples.
• Implemented an adaptive parameter adjustment based on anchor-sample distance to prevent overfitting.

Main Results:

• The proposed method significantly enhances zero-shot adversarial robustness across 16 datasets.
• Achieved an average improvement of 10.83% in top-1 robust accuracy.
• Maintained the original VLMs' zero-shot generalization capabilities effectively.

Conclusions:

• The training-free, zero-shot approach offers a viable solution for improving VLM adversarial robustness.
• This method provides a practical way to defend VLMs against adversarial attacks without extensive retraining.
• The adaptive parameter adjustment is key to preventing overfitting and ensuring sustained performance.