On the choice of parameters of the cost function in nested modular RNN's.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) face vanishing gradient issues in predictions.
• Pipelined Recurrent Neural Networks (PRNNs) offer a modular architecture to mitigate this.
• Existing PRNN cost functions utilize a forgetting factor from recursive least squares (RLS).

Purpose of the Study:

• To propose novel cost function coefficient choices for PRNNs.
• To develop a cost function that provides module output forgetting based on network architecture.
• To enhance PRNN performance in time series prediction.

Main Methods:

• Designing a new cost function for PRNNs with unit norm constraints on coefficients.
• Implementing the modified cost function within the PRNN architecture.
• Evaluating the PRNN's performance on time series prediction tasks.

Main Results:

• The proposed cost function enables forgetting of adjacent module outputs based on PRNN architecture.
• Unit norm constraints ensure each module contributes fully while managing memory.
• The modified PRNN demonstrated superior performance compared to existing PRNN schemes in simulations.

Conclusions:

• The novel cost function effectively addresses memory management in PRNNs.
• This approach enhances the learning process by balancing module contribution and forgetting.
• The modified PRNN shows significant improvements in time series prediction accuracy.