FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers

Area of Science:

• Computer Vision
• Deep Learning Security

Background:

• Vision Transformer (ViT) models are widely used but can lack robustness against adversarial attacks.
• Prioritizing performance in training may compromise model security, raising significant concerns.

Purpose of the Study:

• To investigate parameter-efficient adversarial finetuning for enhancing ViT robustness.
• To develop a method that quickly and effectively improves adversarial resilience using minimal additional parameters.

Main Methods:

• Introduced the LNLoRA module, featuring learnable layer normalization preceding the LoRA module, to address parameter magnitude differences.
• Proposed the FullLoRA framework, integrating LNLoRA modules across ViT components while freezing the pretrained model.
• Implemented adversarial finetuning within the FullLoRA framework for parameter-efficient robustness enhancement.

Main Results:

• FullLoRA demonstrated superior adversarial robustness compared to standard training.
• Achieved robustness comparable to full finetuning using only approximately 5% of learnable parameters.
• Effectively mitigated concerns related to increased storage and training time for adversarial finetuning.

Conclusions:

• The FullLoRA framework offers a parameter-efficient solution for enhancing ViT adversarial robustness.
• This approach significantly improves model security without prohibitive computational overhead.
• FullLoRA presents a practical method for deploying robust ViT models in security-sensitive applications.