Neural network pruning and simultaneous feature and structure selection

Neural network pruning makes large neural networks more portable and smaller neural networks more interpretable with minimal loss of predictive accuracy. We propose a pruning method that reformulates the neural-network LASSO problem as a standard weighted regression or classification problem with a LASSO penalty. We apply this method starting from a dense neural network structure that includes all possible feed-forward networks as subnetworks. This efficiently removes substantial redundancy. To further refine the network, we develop a second step, which cycles over all remaining links, one at a time, removing those that do not sufficiently improve the model fit. We demonstrate the effectiveness and stability of our method for both regression and classification problems in four simulation studies. Our method improves both prediction and interpretability, compared with the original dense neural network, and with state-of-the-art neural network pruning methods. Our method also outperforms the state-of-the-art methods across ten real data examples (five regression and five classification).