Bayesian inference with finitely wide neural networks

Area of Science:

• Machine Learning
• Bayesian Inference
• Deep Learning

Background:

• Machine learning practitioners often model neural networks as Gaussian processes for analytical benefits.
• Finite network widths introduce deviations from ideal Gaussianity, complicating inference.
• Existing methods struggle with accurate modeling of these finite-width effects.

Purpose of the Study:

• To develop a non-Gaussian distribution for modeling outputs of finite-width random neural networks.
• To enable accurate Bayesian regression by deriving non-Gaussian posterior distributions.
• To investigate non-Gaussianity in deep neural networks within a weight space Gaussian process framework.

Main Methods:

• Utilizing multivariate Edgeworth expansion to derive a differential form for non-Gaussian distributions.
• Deriving marginal and conditional properties of the proposed non-Gaussian distribution.
• Analyzing non-Gaussianity in deep neural networks using marginal kernels and small parameters.

Main Results:

• A novel non-Gaussian distribution is proposed for finite neural network outputs.
• The method allows for derivation of non-Gaussian posterior distributions in Bayesian regression.
• Non-Gaussianity in deep Gaussian processes is characterized through specific parameters.

Conclusions:

• The proposed non-Gaussian approach enhances Bayesian inference for finite-width neural networks.
• This work provides a more realistic model for neural network outputs beyond Gaussian assumptions.
• The findings offer improved analytical tractability for complex deep learning models.