Layer wise Scaled Gaussian Priors for Markov Chain Monte Carlo Sampled deep Bayesian neural networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Initialization is crucial for training neural networks and Bayesian neural networks.
• Bayesian neural networks offer advantages like uncertainty estimation and overfitting prevention.

Purpose of the Study:

• To evaluate the performance of Layer-wise Scaled Gaussian Priors in Markov Chain Monte Carlo trained Bayesian neural networks.
• To compare Layer-wise Scaled Gaussian Priors against isotropic priors for efficiency and the cold posterior effect.

Main Methods:

• Experiments were conducted on 8 classification datasets of varying complexity.
• Markov Chain Monte Carlo (MCMC) methods were used to train Bayesian neural networks.
• Layer-wise Scaled Gaussian Priors were compared with Isotropic Gaussian, Cauchy, and Laplace Priors.

Main Results:

• Layer-wise Scaled Gaussian Priors demonstrated more efficient sampling compared to isotropic priors.
• The cold posterior effect was not observed with Isotropic Gaussian or Layer-wise Scaled Priors in small feed-forward networks.

Conclusions:

• Layer-wise Scaled Gaussian Priors offer a significant improvement in the efficiency of MCMC-learned Bayesian neural networks.
• This prior choice mitigates the cold posterior effect, making Bayesian neural networks more reliable.