Extended Kalman Filter-Based Pruning Method for Recurrent Neural Networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Pruning is crucial for improving neural network generalization by reducing complexity.
• Current pruning methods often rely on energy minimization, common in gradient-based learning.
• Extended Kalman Filter (EKF)-based training offers faster convergence than gradient methods for recurrent networks.

Purpose of the Study:

• To introduce and evaluate a novel pruning procedure for recurrent neural networks (RNNs) specifically designed for EKF training.
• To adapt pruning techniques beyond error sensitivity by utilizing posterior probability sensitivity.

Main Methods:

• A pruning procedure for RNNs is developed, leveraging EKF training.
• Posterior probability sensitivity is employed as a metric to determine weight importance, replacing traditional error sensitivity.
• The method is tested on three distinct time series problems: linear prediction, nonlinear system identification, and exchange-rate forecasting.

Main Results:

• The proposed pruning method successfully reduced the number of parameters in the recurrent networks.
• Evaluations across three different prediction tasks demonstrated improved generalization ability of the pruned networks.
• The use of posterior probability sensitivity proved effective for weight importance assessment within the EKF framework.

Conclusions:

• The developed pruning technique offers an effective way to optimize recurrent neural networks trained with EKF.
• This method contributes to more efficient and generalizable neural network models, particularly for time series analysis.
• Posterior probability sensitivity is a viable alternative to error sensitivity for pruning in EKF-based recurrent network training.