Spike-and-Slab Shrinkage Priors for Structurally Sparse Bayesian Neural Networks.

Area of Science:

• Deep learning and artificial intelligence
• Computational neuroscience and machine learning
• Bayesian inference and statistical modeling

Background:

• Deep learning models often suffer from high complexity and computational inefficiency due to overparameterization.
• Sparse deep learning offers a solution by reducing network size for improved performance and efficiency.
• Model compression techniques are crucial for deploying deep neural networks in resource-constrained environments.

Purpose of the Study:

• To explore Lasso and Horseshoe shrinkage techniques for compressing Bayesian neural networks (BNNs).
• To propose and develop structurally sparse BNNs using novel spike-and-slab priors.
• To establish theoretical guarantees for the proposed models and demonstrate their empirical effectiveness.

Main Methods:

• Implementation of spike-and-slab group Lasso (SS-GL) and spike-and-slab group Horseshoe (SS-GHS) priors for structured sparsity.
• Development of computationally tractable variational inference methods, including continuous relaxation of Bernoulli variables.
• Theoretical analysis of variational posterior contraction rates based on network topology and weights.

Main Results:

• The proposed structurally sparse BNNs achieve competitive prediction accuracy compared to baseline models.
• Significant model compression is achieved through systematic pruning of excessive network nodes.
• Reduced inference latency and improved computational efficiency were empirically demonstrated.

Conclusions:

• Structurally sparse BNNs with SS-GL and SS-GHS priors offer an effective approach for deep learning model compression.
• The developed variational inference methods provide a computationally tractable solution for training these sparse models.
• The findings highlight the potential of structured sparsity in BNNs for achieving efficient and accurate deep learning systems.