MDAPT: Multi-Modal Depth Adversarial Prompt Tuning to Enhance the Adversarial Robustness of Visual Language Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Large visual language models (VLMs) like CLIP excel in performance but are vulnerable to adversarial examples.
• Existing methods often struggle to maintain robustness against sophisticated attacks.

Purpose of the Study:

• To investigate the accuracy and robustness of VLMs from a multi-modal perspective.
• To propose a novel fine-tuning method to enhance VLM resilience against adversarial attacks.

Main Methods:

• Developed Multi-modal Depth Adversarial Prompt Tuning (MDAPT), a fine-tuning technique.
• MDAPT guides visual prompt generation using text prompts for improved VLM performance.
• Conducted extensive experiments on three datasets under adversarial conditions.

Main Results:

• Achieved significant performance improvements on three datasets (ϵ=4/255).
• MDAPT increased accuracy by an average of 17.84% and robustness by 10.85% compared to manual prompts.
• Demonstrated substantial gains (32.16% accuracy, 21.00% robustness) under three different attack methods with efficient settings.

Conclusions:

• MDAPT effectively enhances the accuracy and robustness of VLMs against adversarial examples.
• The proposed multi-modal approach offers a promising direction for developing more resilient AI systems.
• MDAPT provides a significant advantage over traditional prompt design strategies in adversarial settings.