Graph Reparameterizations for Enabling 1000+ Monte Carlo Iterations in Bayesian Deep Neural Networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Uncertainty estimation in deep models is crucial for real-world applications.
• Existing methods using Gaussian formulations may be insufficient.
• Monte Carlo (MC) sampling for KL divergence is computationally expensive and scales poorly with data and model dimensions.

Purpose of the Study:

• To develop a framework for analyzing computation graphs in uncertainty estimation.
• To identify probability families where computation graph size is independent or weakly dependent on MC samples.
• To improve the efficiency and scalability of uncertainty estimation in deep learning.

Main Methods:

• Constructing a framework to describe computation graphs.
• Identifying specific probability families for efficient MC sampling.
• Empirical evaluation on large computer vision architectures.

Main Results:

• The proposed framework allows for computation graphs independent of MC sample size.
• Identified probability families enable more scalable uncertainty estimation.
• Empirical results show performance gains in confident accuracy, training stability, memory, and training time for larger models.

Conclusions:

• The developed framework offers a more efficient approach to uncertainty estimation in deep learning.
• This method addresses the scalability limitations of traditional MC sampling.
• The findings have significant implications for improving the reliability and performance of deep models in practice.