Neural-network design for small training sets of high dimension

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Statistics

Background:

• Designing neural networks with limited data (input dimension d comparable to training set size n) often leads to overfitting.
• Standard methods result in high-complexity networks, causing poor generalization to new data.

Purpose of the Study:

• To develop a statistically-based methodology for designing neural networks that ensures good generalization performance even with limited training data.
• To address the challenge of selecting appropriate network architecture and input variables when d ≈ n.

Main Methods:

• Feature selection using nonparametric difference-based variance estimation.
• Network architecture selection via projection pursuit regression (PPR) and slicing inverse regression (SIR).
• Retraining the selected network without PPR/SIR parameters to achieve moderate complexity.

Main Results:

• The proposed methodology yields a network of moderate complexity (parameters << n).
• This approach is expected to significantly improve generalization performance on unseen data.
• Demonstrated effectiveness through short-term electric power demand forecasting.

Conclusions:

• The statistically-based methodology effectively designs neural networks for scenarios with limited training data.
• Careful selection of input variables and network architecture is crucial for good generalization.
• The method offers a robust solution for practical applications like demand forecasting.