Efficient Online Learning Algorithms Based on LSTM Neural Networks

Area of Science:

• Machine Learning
• Deep Learning
• Statistical Modeling

Background:

• Online nonlinear regression presents challenges in real-time parameter estimation.
• Existing methods often lack efficiency or guaranteed convergence for complex models.

Purpose of the Study:

• To develop novel regression structures using Long Short-Term Memory (LSTM) networks for online nonlinear regression.
• To introduce efficient and effective online training methods for these novel structures.
• To demonstrate the superiority of LSTM-based approaches over conventional methods.

Main Methods:

• Developed LSTM-based regression structures and formulated them in a state-space form.
• Implemented particle filtering (PF)-based updates for efficient training.
• Also provided stochastic gradient descent and extended Kalman filter-based updates.
• Introduced a Gated Recurrent Unit (GRU)-based alternative.

Main Results:

• PF-based training guarantees convergence to optimal parameter estimation under specific conditions.
• Achieved performance comparable to first-order gradient methods with controlled particle counts.
• LSTM-based approach demonstrated superiority in sequential prediction tasks on real-world datasets.
• Significant performance improvements observed compared to conventional methods.

Conclusions:

• Novel LSTM and GRU-based regression structures offer efficient and effective online nonlinear regression.
• Particle filtering provides a robust and computationally efficient training mechanism.
• The proposed methods significantly outperform traditional approaches in sequential prediction tasks.