An Adaptive Empirical Bayesian Method for Sparse Deep Learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Deep learning models often require significant computational resources and are vulnerable to adversarial attacks.
• Achieving sparsity in deep neural networks is crucial for model compression and efficiency.
• Existing Bayesian methods for sparsity can be computationally intensive and difficult to optimize.

Purpose of the Study:

• To develop a novel adaptive empirical Bayesian (AEB) method for sparse deep learning.
• To ensure sparsity through self-adaptive spike-and-slab priors.
• To improve the performance, compression, and adversarial robustness of deep learning models.

Main Methods:

• The proposed AEB method alternatively samples from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC).
• Hyperparameters are smoothly optimized using stochastic approximation (SA).
• Convergence of the method to the asymptotically correct distribution is proven under mild conditions.

Main Results:

• State-of-the-art performance achieved on MNIST and Fashion MNIST datasets using shallow convolutional neural networks (CNNs).
• State-of-the-art compression performance demonstrated on CIFAR10 dataset with Residual Networks.
• Improved resistance to adversarial attacks observed in empirical applications.

Conclusions:

• The novel AEB method offers a powerful and efficient approach for sparse deep learning.
• The method achieves superior performance, compression, and robustness compared to existing techniques.
• This work has significant implications for developing more efficient and secure deep learning models.