Gradient Regularization as Approximate Variational Inference.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Bayesian neural networks (BNNs) offer probabilistic predictions but often require computationally intensive methods like stochastic sampling.
• Variational inference (VI) is a common technique for approximating BNN posteriors, but standard methods can be sensitive to convergence issues.
• Maximum a posteriori (MAP) inference provides a point estimate but lacks uncertainty quantification.

Purpose of the Study:

• To develop an efficient and accurate method for approximating the evidence lower bound (ELBO) in Bayesian neural networks.
• To improve the test performance and calibration of BNNs compared to existing inference techniques.
• To provide a robust benchmarking methodology for variational inference in BNNs.

Main Methods:

• Developed Variational Laplace, a novel inference technique for BNNs.
• Utilized a local approximation of the likelihood's curvature to estimate the ELBO.
• Introduced a simple, non-stochastic objective function: log-likelihood + weight-decay + squared-gradient regularizer.
• Benchmarked Variational Laplace against MAP inference and standard sampling-based VI.

Main Results:

• Variational Laplace achieved superior test performance and lower expected calibration errors compared to MAP and standard VI.
• The method avoids stochastic sampling of neural network weights, simplifying computation.
• Demonstrated that careful tuning of learning rates for variance parameters can prevent early stopping in standard VI.

Conclusions:

• Variational Laplace presents an effective and computationally efficient alternative for Bayesian neural network inference.
• The proposed method enhances predictive accuracy and model calibration.
• Highlights the importance of proper convergence monitoring and parameter tuning in variational inference benchmarking.