On the Robustness of Bayesian Neural Networks to Adversarial Attacks

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence Security

Background:

• Deep learning adoption in safety-critical applications is hindered by vulnerability to adversarial attacks.
• Current methods struggle to train robust deep learning models against these threats.
• Understanding the geometric properties of adversarial attacks is crucial.

Purpose of the Study:

• To analyze the geometry of adversarial attacks in the over-parameterized limit for Bayesian neural networks (BNNs).
• To demonstrate the inherent robustness of BNN posteriors to gradient-based adversarial attacks.
• To theoretically and empirically validate BNNs' resilience against various attack types.

Main Methods:

• Analysis of adversarial attack geometry in the over-parameterized limit for BNNs.
• Leveraging the convergence of infinitely-wide BNNs to Gaussian processes (GPs).
• Theoretical proof of vanishing expected loss gradient with respect to BNN posterior distribution.
• Experimental validation using Hamiltonian Monte Carlo and variational inference on benchmark datasets.

Main Results:

• Adversarial attack vulnerability in the limit stems from data distribution degeneracy (lower-dimensional submanifolds).
• BNN posteriors are robust to gradient-based adversarial attacks in the over-parameterized limit.
• Expected loss gradient vanishes for BNN posteriors, even if individual networks do not.
• Empirical evidence shows BNNs maintain high accuracy and robustness to gradient-based and gradient-free attacks.

Conclusions:

• The geometric properties of data distributions in the over-parameterized limit explain BNN robustness.
• BNNs offer a promising solution for developing deep learning models resilient to adversarial attacks.
• BNNs demonstrate potential for safe deployment in safety-critical AI applications.