Estimates on compressed neural networks regression.

Area of Science:

• Machine Learning
• Neural Networks
• Statistical Learning Theory

Background:

• Overfitting is a common problem in neural networks when the number of neural elements (n) exceeds the sample size (m).
• This occurs because there are more parameters than data points, leading to models that do not generalize well.

Purpose of the Study:

• To propose a novel method for overcoming the overfitting problem in feedforward neural networks (FNNs).
• To reduce the number of neural elements by employing compressed projection without requiring the Restricted Isometric Property (RIP).

Main Methods:

• Utilizing probability inequalities and approximation properties specific to FNNs.
• Applying compressed projection to solve the FNNs regression learning algorithm in a compressed domain.
• Employing covering number theory to estimate the excess error.

Main Results:

• Solving the FNNs regression learning algorithm in the compressed domain reduces sample error.
• The approximation error increases but remains controlled.
• An upper bound for the excess error is derived.

Conclusions:

• Compressed projection offers an effective strategy to mitigate overfitting in FNNs.
• This approach balances sample error reduction with controlled approximation error, enhancing model performance.
• The theoretical framework provides a bound on the excess error, validating the method.