Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Pre-trained vision-language models like CLIP excel at zero-shot tasks but are vulnerable to adversarial examples.
• Adversarial perturbations can alter text-guided attention mechanisms in these models.
• Existing methods struggle with maintaining generalization while improving robustness.

Purpose of the Study:

• To enhance the adversarial robustness of vision-language models without sacrificing generalization.
• To address the issue of attention focusing on irrelevant features in adversarial scenarios.
• To improve zero-shot robust accuracy in challenging datasets.

Main Methods:

• Proposed Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR) with Local Attention Refinement and Global Attention Constraint modules.
• TGA-ZSR aligns attention maps from adversarial and clean examples and constrains attention on clean data.
• Introduced Complementary Text-Guided Attention (Comp-TGA) integrating class and non-class prompt-guided attention for comprehensive foreground representation.

Main Results:

• TGA-ZSR demonstrated significant improvements in zero-shot robust accuracy.
• Comp-TGA further enhanced robustness by utilizing complementary attention mechanisms.
• Experiments across 16 datasets showed TGA-ZSR and Comp-TGA achieved 9.58% and 11.95% improvements, respectively, over state-of-the-art methods.

Conclusions:

• TGA-ZSR and Comp-TGA are effective strategies for improving adversarial robustness in vision-language models.
• Complementary attention mechanisms offer a promising direction for robust zero-shot learning.
• The proposed methods maintain generalization while significantly boosting performance on adversarial examples.