LSTM: A Search Space Odyssey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Long Short-Term Memory (LSTM) networks are advanced recurrent neural networks.
• Various LSTM architectures have been developed since 1995.
• These networks are state-of-the-art for many machine learning tasks, driving interest in their components.

Purpose of the Study:

• To conduct a large-scale analysis of eight LSTM variants.
• To evaluate the utility of different computational components within LSTM architectures.
• To compare LSTM variants on speech recognition, handwriting recognition, and music modeling tasks.

Main Methods:

• Performed a large-scale analysis of 5400 experimental runs across eight LSTM variants.
• Optimized hyperparameters for each variant and task using random search.
• Assessed hyperparameter importance using functional ANalysis Of VAriance (fANOVA).

Main Results:

• No LSTM variant significantly outperformed the standard LSTM architecture.
• The forget gate and output activation function were identified as the most critical LSTM components.
• Studied hyperparameters demonstrated near-complete independence.

Conclusions:

• Standard LSTM architecture is highly effective for speech, handwriting, and music modeling.
• Forget gate and output activation function are key to LSTM performance.
• Derived guidelines for efficient hyperparameter adjustment in LSTMs.