Nearly Optimal Learning Using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization With Lipschitz...

Area of Science:

• Machine Learning
• Statistical Learning Theory
• Deep Learning

Background:

• Empirical risk minimization is a standard approach for statistical estimation.
• Lipschitz loss functions are commonly used in regression and classification.
• Deep neural networks often struggle with overfitting and parameter efficiency.

Purpose of the Study:

• To propose a Sparse Deep ReLU Network (SDRN) estimator for regression functions.
• To establish nonasymptotic excess risk bounds for the SDRN estimator.
• To analyze the convergence rate and parameter complexity of the SDRN.

Main Methods:

• Developing a sparse deep ReLU network architecture.
• Utilizing regularized empirical risk minimization with a Lipschitz loss function.
• Deriving nonasymptotic excess risk bounds for Sobolev spaces with mixed derivatives.

Main Results:

• The SDRN estimator achieves a nearly optimal minimax convergence rate, comparable to 1D nonparametric regression.
• The convergence rate is logarithmic in feature dimension when fixed, and slightly slower when dimension grows with sample size.
• The depth of the SDRN grows logarithmically with sample size, while nodes/weights grow polynomially.

Conclusions:

• The proposed SDRN estimator effectively estimates regression functions.
• SDRN overcomes overfitting issues in conventional feedforward networks.
• SDRN offers improved depth and parameter efficiency for enhanced regression performance.