Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality.

Area of Science:

• Machine Learning
• Statistics
• Computer Science

Background:

• Deep neural networks (DNNs) with sparsity constraints offer optimal convergence rates for regression and classification.
• Practical application is hindered by the need for prior model knowledge and computational challenges associated with discrete sparsity constraints.

Purpose of the Study:

• To propose a novel penalized estimation method for sparse DNNs that addresses the limitations of traditional sparsity constraints.
• To establish theoretical guarantees for the proposed method and demonstrate its computational efficiency.

Main Methods:

• Developed a penalized estimation approach for sparse DNNs, avoiding discrete sparsity constraints.
• Established an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator.
• Derived convergence rates for various learning tasks, including nonparametric regression.

Main Results:

• The proposed sparse-penalized DNN estimator achieves optimal convergence rates.
• The estimator adaptively attains minimax convergence rates for nonparametric regression problems.
• An efficient gradient-based optimization algorithm was developed, ensuring monotonic reduction of the objective function.

Conclusions:

• The novel penalized estimation method effectively resolves practical and computational issues associated with sparsity constraints in DNNs.
• The method demonstrates strong theoretical performance, achieving adaptive minimax rates and efficient computation for sparse DNNs.