Reliable uncertainty estimates in deep learning with efficient Metropolis-Hastings algorithms

Area of Science:

• Machine Learning
• Computational Statistics

Background:

• Data-driven models require reliable uncertainty estimates for robust decision-making.
• Bayesian neural networks (BNNs) provide uncertainty quantification for deep learning.
• Traditional sampling methods like Hamiltonian Monte Carlo (HMC) are computationally expensive.

Purpose of the Study:

• To develop computationally efficient methods for uncertainty estimation in BNNs.
• To integrate lightweight Metropolis-Hastings steps into stochastic gradient HMC.
• To improve prediction accuracy and calibration while maintaining computational feasibility.

Main Methods:

• Incorporating noisy Metropolis-Hastings acceptance steps into deep neural networks.
• Utilizing batched training samples for acceptance steps to reduce computational load.
• Developing stochastic gradient-driven trajectories inspired by the Hamiltonian ensemble concept.

Main Results:

• Achieved up to 5.8% improvement in prediction accuracy over deterministic models.
• Improved prediction accuracy by up to 4.3% compared to standard Bayesian approaches.
• Maintained prediction calibration and demonstrated efficiency with reduced ensemble sizes.

Conclusions:

• The proposed methods combine the efficiency of stochastic gradients with regularization effects.
• These techniques offer strong performance and reliable uncertainty estimates despite introducing sampling bias.
• The findings suggest a practical approach to enhancing BNNs for real-world applications.