A New Delay Connection for Long Short-Term Memory Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural network (NN) learning heavily relies on connection mechanisms for information flow.
• Existing Long Short-Term Memory (LSTM) units face challenges with gradient problems and maintaining information flow.
• Enhancing recurrent units is crucial for improving learning capabilities.

Purpose of the Study:

• To introduce a novel Delay Connected LSTM (DCLSTM) unit to enhance recurrent neural network performance.
• To improve information flow and back-propagation of error signals in deep learning models.
• To evaluate the effectiveness of DCLSTM against state-of-the-art recurrent models.

Main Methods:

• A new delay connection mechanism was integrated into the LSTM unit, creating the DCLSTM.
• The DCLSTM model was evaluated on sequence classification tasks, comparing it with other recurrent models.
• Performance was further assessed using stacked DCLSTM layers on the Penn Treebank (PTB) language modeling task.

Main Results:

• The DCLSTM model demonstrated superior performance, achieving higher accuracy and F1 scores in sequence classification.
• On the PTB language modeling task, DCLSTM achieved lower perplexity (PPL)/bit-per-character (BPC) compared to standard LSTM.
• The DCLSTM model exhibited more stable and efficient learning processes.

Conclusions:

• The proposed delay connection in DCLSTM enhances learning capabilities without introducing additional parameters.
• DCLSTM effectively bridges error signals to previous time steps, combating vanishing gradients.
• DCLSTM represents a significant advancement in recurrent neural network architecture for complex sequence tasks.