Learning Sparse Deep Neural Networks with a Spike-and-Slab Prior

Area of Science:

• Machine Learning
• Artificial Intelligence
• Statistics

Background:

• Deep neural networks (DNNs) demonstrate significant success but suffer from over-parameterization, leading to high computational costs, memory demands, and poor interpretability.
• Over-parameterized DNNs often exhibit mis-calibration, impacting their reliability in real-world applications.

Purpose of the Study:

• To investigate the benefits of sparsity in DNNs within a Bayesian framework.
• To establish theoretical guarantees for Bayesian DNNs using spike-and-slab priors.
• To demonstrate the practical advantages of sparse DNNs in various machine learning tasks.

Main Methods:

• Employing a Bayesian framework for deep neural networks.
• Utilizing spike-and-slab priors to induce sparsity.
• Establishing posterior consistency and structure selection consistency for Bayesian DNNs.
• Evaluating performance on high-dimensional nonlinear variable selection, network compression, and model calibration.

Main Results:

• Demonstrated posterior consistency and structure selection consistency for Bayesian DNNs with spike-and-slab priors.
• Showcased improved prediction accuracy through sparsity in DNNs.
• Validated the effectiveness of sparsity in enhancing model calibration.
• Provided numerical evidence across diverse applications including variable selection and network compression.

Conclusions:

• Sparsity is crucial for enhancing both prediction accuracy and calibration in DNNs.
• The Bayesian approach with spike-and-slab priors provides a robust framework for developing and analyzing sparse DNNs.
• Sparse DNNs offer a computationally efficient and more reliable alternative to heavily parameterized models.