Consistent Sparse Deep Learning: Theory and Computation.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Deep neural networks (DNNs) are powerful but often over-parameterized, leading to training and interpretation challenges.
• Over-parameterization in DNNs hinders efficient prediction and increases complexity.

Purpose of the Study:

• To develop a frequentist-like method for learning sparse DNNs.
• To provide theoretical justifications for the proposed sparse DNN learning method within a Bayesian framework.
• To advance interpretable machine learning through efficient network compression and variable selection.

Main Methods:

• Proposed a frequentist-like approach to learn sparse deep neural networks (DNNs).
• Justified the method's consistency under the Bayesian framework, ensuring theoretical guarantees.
• Utilized a Laplace approximation-based marginal posterior inclusion probability for structure determination.
• Employed Bayesian evidence for eliciting sparse DNNs via optimization methods like stochastic gradient descent.

Main Results:

• Learned sparse DNNs with at most O(n/log(n)) connections.
• Established posterior consistency and variable selection consistency for sparse DNNs.
• Demonstrated computational efficiency compared to standard Bayesian methods for large-scale DNNs.
• Achieved strong performance in large-scale network compression and high-dimensional nonlinear variable selection.

Conclusions:

• The proposed method effectively learns sparse DNNs with theoretical guarantees.
• It offers a computationally efficient alternative for large-scale applications.
• The approach contributes to advancing interpretable machine learning by enabling effective network compression and variable selection.