On the regularization of forgetting recursive least square.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Neural Networks

Background:

• Feedforward neural networks require regularization techniques to prevent overfitting and improve generalization.
• Traditional methods like weight decay are effective but can be computationally intensive.
• Recursive Least Square (RLS) is an adaptive filtering technique used in training neural networks.

Purpose of the Study:

• To investigate the regularization effects of the forgetting recursive least square (FRLS) training technique on feedforward neural networks.
• To compare the FRLS technique with the standard weight decay method and the conventional RLS method.

Main Methods:

• Derivation of results from equations for expected prediction error and expected training error.
• Analysis of the FRLS technique's impact on model complexity and prediction error.
• Comparative study against the weight decay method and standard RLS.

Main Results:

• The FRLS technique demonstrates an effect identical to the simple weight decay method, offering an online approach for regularization.
• Under specific conditions, FRLS leads to reduced model complexity and lower expected prediction error compared to standard RLS.
• FRLS provides an alternative online realization of the weight decay effect.

Conclusions:

• The FRLS training technique is a viable online regularization method for feedforward neural networks.
• FRLS offers advantages over standard RLS in terms of model complexity and prediction accuracy.
• This study establishes FRLS as an effective alternative for implementing weight decay.