Two-Stage Orthogonal Least Squares Methods for Neural Network Construction.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Mathematics

Background:

• Neural networks can often be represented as linear-in-the-parameters models.
• Training these models involves selecting a compact subset of parameters, a task typically handled by subset selection algorithms.
• Existing methods like forward selection can lead to suboptimal models and local minima.

Purpose of the Study:

• To propose novel unified two-stage orthogonal least squares methods for training linear-in-the-parameters neural networks.
• To enhance model compactness and generalization performance compared to existing recursive algorithms.
• To reduce computational demand through simplified relationships and efficient refinement schemes.

Main Methods:

• Development of unified two-stage orthogonal least squares (OLS) methods.
• Derivation of a simplified relationship between forward and backward stages leveraging OLS properties to avoid redundant computations.
• Introduction of a new term exchanging scheme for backward model refinement.
• Implementation of forward and backward subset selection procedures guided by the error reduction ratio criterion.

Main Results:

• The proposed methods demonstrate improved model compactness.
• The techniques show enhanced efficiency in subset selection for linear neural networks.
• Experimental results validate the superior performance compared to popular existing methods.
• Reduced computational demand is achieved through algorithmic innovations.

Conclusions:

• Unified two-stage orthogonal least squares methods offer an effective and efficient approach for training linear-in-the-parameters neural networks.
• The proposed techniques overcome limitations of traditional forward selection and recursive algorithms.
• The study highlights the benefits of leveraging orthogonal properties for streamlined model selection and refinement.