Online Training of LSTM Networks in Distributed Systems for Variable Length Data Sequences

Area of Science:

• Machine Learning
• Distributed Systems
• Signal Processing

Background:

• Online training of Long Short-Term Memory (LSTM) architectures is crucial for real-time data analysis in distributed networks.
• Existing methods often face challenges in efficiency and convergence within decentralized computational environments.
• LSTM networks are widely used for sequence data regression but require effective distributed training strategies.

Purpose of the Study:

• To investigate and develop an effective online training algorithm for LSTM architectures in a distributed network setting.
• To propose a novel distributed particle filtering (DPF)-based training algorithm for LSTM-based online regression.
• To compare the DPF approach with a distributed extended Kalman filtering (DEKF) method for performance evaluation.

Main Methods:

• Developed a generic LSTM-based regression structure for individual nodes in a distributed network.
• Formulated LSTM equations into a nonlinear state-space model for each node.
• Introduced a distributed particle filtering (DPF) algorithm and a distributed extended Kalman filtering (DEKF) algorithm for training.

Main Results:

• The DPF-based training algorithm demonstrated convergence to optimal LSTM coefficients in the mean square error sense under specific conditions.
• Achieved performance comparable to first-order gradient-based methods in terms of communication and computational complexity.
• Simulated and real-life examples showed significant performance improvements over existing state-of-the-art methods.

Conclusions:

• The proposed DPF algorithm offers an efficient and effective solution for online LSTM training in distributed networks.
• DPF provides a robust method for achieving high performance in distributed online regression tasks.
• This approach advances the capabilities of distributed machine learning for complex sequential data.