Gating Revisited: Deep Multi-Layer RNNs That can be Trained

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) are crucial for sequence modeling.
• Deep RNN architectures face challenges with parameter inefficiency and gradient instability (vanishing/exploding gradients).
• Existing recurrent cells like LSTM and GRU can be computationally expensive and prone to gradient issues when stacked.

Purpose of the Study:

• To propose a novel recurrent cell, the STAckable Recurrent (STAR) cell.
• To address the limitations of parameter cost and gradient propagation in deep RNNs.
• To enhance the performance and computational efficiency of sequence modeling.

Main Methods:

• Investigated gradient propagation in multi-layer RNNs, analyzing vertical gradient magnitudes.
• Designed a new gated cell (STAR) to preserve gradient magnitude during training.
• Validated the STAR cell on diverse sequence modeling tasks.

Main Results:

• The STAR cell demonstrates greater robustness against vanishing or exploding gradients compared to LSTM and GRU.
• STAR cells require fewer parameters and computational resources.
• Enabled the successful training of deeper recurrent architectures.
• Achieved improved performance on sequence modeling tasks.

Conclusions:

• The STAR cell provides a more efficient and stable alternative for building deep recurrent neural networks.
• STAR facilitates the development of more powerful and computationally feasible sequence models.
• This innovation offers a promising direction for advancing recurrent neural network architectures.