Back propagation through adjoints for the identification of nonlinear dynamic systems using recurrent neural models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Backpropagation is a fundamental algorithm for training artificial neural networks.
• Efficient gradient evaluation in recurrent neural networks (RNNs) with arbitrary interconnections presents challenges.
• Existing methods may suffer from delays in gradient computation.

Purpose of the Study:

• To reinvestigate backpropagation for efficient gradient evaluation in recurrent subsystems.
• To develop faster and more robust backpropagation variants.
• To analyze convergence properties of novel training schemes.

Main Methods:

• Reinvestigation of backpropagation through the adjoint model.
• Development of accelerated backpropagation to eliminate delay.
• Proposal of predictive and targeted backpropagation, inspired by Lyapunov stability and adjoint methods.
• Introduction of predictive backpropagation with filtering for state updates.
• Lyapunov analysis for convergence of feedforward model updates.

Main Results:

• Gradient evaluation requires backpropagation through the adjoint model, potentially with a delay.
• Accelerated backpropagation removes the delay for faster training.
• Predictive and targeted backpropagation offer novel approaches for weight updates.
• Predictive backpropagation with filtering ensures convergence.
• Sufficient condition for convergence: backpropagating time steps equal to the system order.
• Convergence analysis of online batch and sample-wise updates in feedforward networks.

Conclusions:

• The adjoint model is crucial for efficient gradient computation in recurrent systems.
• Novel backpropagation variants offer improved efficiency and convergence guarantees.
• The proposed methods advance the training of complex recurrent neural networks.