Initializing LSTM internal states via manifold learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Dynamical Systems

Background:

• Recurrent neural networks, particularly Long Short-Term Memory (LSTM) networks, are powerful tools for time series analysis.
• Initializing the internal states of LSTMs consistently with input data is crucial for accurate modeling.
• Partially observed dynamical systems pose challenges for traditional system identification methods.

Purpose of the Study:

• To present a novel approach for initializing LSTM internal states based on learning an intrinsic data manifold.
• To demonstrate how this initialization method ensures consistency with initial observed input data.
• To show that learning the data manifold can transform partially observed dynamics into fully observed ones.

Main Methods:

• Learning an intrinsic data manifold from observed input data.
• Utilizing the concept of generalized synchronization to define converged internal states as a function on the learned manifold.
• Applying the approach to a partially observed chemical model system.

Main Results:

• The dimension of the learned manifold dictates the required length of input time series for consistent initialization.
• Initializing LSTM internal states using the learned manifold yields visibly improved performance in a chemical model system.
• Learning the data manifold enables the transformation of partially observed dynamics into fully observed ones.

Conclusions:

• The proposed manifold learning approach provides a robust method for LSTM internal state initialization.
• This technique enhances the performance of LSTMs in modeling time series data, especially for partially observed systems.
• The method offers new pathways for the identification of nonlinear dynamical systems by facilitating the transition from partial to full observation.