S$^{2}$2O: Enhancing Adversarial Training With Second-Order Statistics of Weights

Area of Science:

• Machine Learning
• Deep Learning
• Optimization Theory

Background:

• Adversarial training enhances deep neural network (DNN) robustness against perturbations.
• Current methods often rely on the unrealistic assumption of statistical independence of model weights.
• Gradient descent methods like SGD are standard for optimizing DNN weights.

Purpose of the Study:

• To propose a novel approach for enhancing adversarial training using Second-Order Statistics Optimization (S$^{2}$2O).
• To relax the assumption of statistical independence of weights in PAC-Bayesian frameworks.
• To derive an improved PAC-Bayesian robust generalization bound.

Main Methods:

• Treating model weights as random variables for optimization.
• Developing and applying Second-Order Statistics Optimization (S$^{2}$2O) over model weights.
• Relaxing the statistical independence assumption in PAC-Bayesian analysis.

Main Results:

• Derived an improved PAC-Bayesian robust generalization bound.
• Demonstrated that optimizing second-order weight statistics tightens the generalization bound.
• Empirically validated that S$^{2}$2O enhances DNN robustness and generalization.

Conclusions:

• S$^{2}$2O offers a principled way to improve adversarial training.
• The method enhances both robustness and generalization of DNNs.
• S$^{2}$2O effectively complements existing state-of-the-art adversarial training techniques.