Fast Adversarial Training With Adaptive Step Size

Area of Science:

• Machine Learning
• Computer Vision
• Cybersecurity

Background:

• Adversarial training is effective against adversarial attacks but suffers from slow training times, hindering scalability to large datasets.
• Current acceleration methods use single-step attacks, risking catastrophic overfitting and loss of robustness.
• Catastrophic overfitting is linked to instance-specific characteristics, particularly input gradient norms.

Purpose of the Study:

• To investigate the instance-dependent nature of catastrophic overfitting in adversarial training.
• To propose a novel method, Adversarial Training with Adaptive Step size (ATAS), to address catastrophic overfitting.
• To demonstrate the effectiveness of ATAS in improving training speed and robust accuracy.

Main Methods:

• Analyzing the relationship between instance gradient norms and catastrophic overfitting.
• Developing ATAS, which employs an instance-wise adaptive step size inversely proportional to the input gradient norm.
• Conducting theoretical analysis to prove faster convergence of ATAS compared to non-adaptive methods.

Main Results:

• Empirical validation on CIFAR10, CIFAR100, and ImageNet datasets.
• ATAS effectively mitigates catastrophic overfitting across various adversarial budgets.
• ATAS achieves superior robust accuracy compared to existing methods.

Conclusions:

• Catastrophic overfitting in adversarial training is instance-dependent and related to gradient norms.
• ATAS offers a scalable and effective solution to catastrophic overfitting, enhancing adversarial robustness.
• The proposed method provides a promising direction for efficient and robust adversarial training.