NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning Security

Background:

• Vision-Language Models (VLMs) excel at joint visual-textual understanding but are vulnerable to adversarial attacks.
• Existing defenses like Adversarial Prompt Tuning (AdvPT) improve robustness but can be enhanced.
• Adversarial perturbations pose significant security risks to VLMs.

Purpose of the Study:

• To introduce the Neural Augmentor framework for Multi-modal Adversarial Prompt Tuning (NAP-Tuning).
• To enhance adversarial robustness in VLMs through multi-modal, multi-layer feature purification.
• To develop an adaptive defense mechanism for identifying and rectifying adversarial perturbations.

Main Methods:

• Developed a comprehensive multi-modal (text and visual) and multi-layer prompting framework (NAP-Tuning).
• Implemented a Neural Augmentor approach with TokenRefiners for feature-level purification via residual connections.
• Conducted experiments across various datasets and attack types, including AutoAttack.

Main Results:

• NAP-Tuning significantly outperforms existing adversarial robustness methods.
• Achieved substantial improvements over baselines under AutoAttack (32.3% on ViT-B16, 31.3% on ViT-B32).
• Maintained competitive clean accuracy while enhancing adversarial defense.

Conclusions:

• Internal feature-level intervention is effective for prompt tuning in adversarial robustness.
• NAP-Tuning offers an adaptive defense by rectifying perturbations within embedding spaces.
• This approach moves beyond input-side alignment for more robust VLMs.