A generalized LSTM-like training algorithm for second-order recurrent neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Long Short-Term Memory (LSTM) networks are powerful for sequential data but have limited training applicability.
• Original LSTM training algorithms offer spatial and temporal locality but restrict network architectures.

Purpose of the Study:

• Introduce the generalized long short-term memory (LSTM-g) training algorithm.
• Expand the applicability of LSTM-like locality to diverse second-order recurrent neural network architectures.
• Improve the performance and versatility of gradient-based training for recurrent networks.

Main Methods:

• Developed the LSTM-g training algorithm where all units share identical activation and learning instructions.
• Applied LSTM-g to LSTM architectures with peephole connections, utilizing an additional error source.
• Demonstrated training task-specific recurrent networks using LSTM-g.

Main Results:

• LSTM-g provides LSTM-like locality across a wider range of network architectures.
• LSTM-g with peephole connections can outperform the original LSTM algorithm.
• Task-specific recurrent networks trained with LSTM-g showed superior performance compared to single-layer networks.

Conclusions:

• LSTM-g significantly broadens the applicability of spatially and temporally local training algorithms.
• The generalized algorithm enhances the potential for improved performance in recurrent neural networks.
• LSTM-g facilitates the engineering of more effective recurrent networks for specific tasks.