Scalable Moment Propagation and Analysis of Variational Distributions for Practical Bayesian Deep Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Statistics

Background:

• Bayesian deep learning is crucial for quantifying predictive uncertainty.
• Variational inference (VI) using Monte Carlo (MC) sampling is computationally expensive.
• Moment propagation (MP) offers a potentially faster alternative but faces challenges in deep models.

Purpose of the Study:

• To develop a fast and reliable MP-based Bayesian deep learning method.
• To address the challenges of training deep models with MP, specifically variance in activations.
• To investigate the impact of different variational distributions on MP's performance and calibration.

Main Methods:

• Introduced an extended batch normalization layer for random variables to manage activation variance in deep MP models.
• Investigated the treatment of moments across various variational distributions to assess prediction uncertainty quality.
• Conducted experiments on regression and classification tasks to evaluate the proposed method.

Main Results:

• The MP-based method achieved predictive performance equivalent to MC-based methods in regression tasks.
• Extended batch normalization enabled the training of MP-based deep models for classification tasks.
• The MP-based approach demonstrated 2.0-2.8 times faster inference than MC-based methods while preserving predictive accuracy.

Conclusions:

• The proposed MP-based Bayesian deep learning method is fast and reliable.
• The extended batch normalization is effective for training deep MP models.
• This work facilitates efficient and well-calibrated uncertainty estimation for reliability-aware applications.