Gated Orthogonal Recurrent Units: On Learning to Forget.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) are crucial for sequential data processing.
• Existing RNN architectures like LSTMs and GRUs have limitations in managing long-term dependencies.
• Unitary and orthogonal RNNs offer stability but struggle with forgetting irrelevant information.

Purpose of the Study:

• To introduce a novel RNN model combining memory retention and selective forgetting capabilities.
• To address the limitations of existing RNNs in handling long-term dependencies and noisy data.
• To improve performance on complex sequential tasks requiring both remembering and forgetting.

Main Methods:

• Developed a novel RNN architecture by extending restricted orthogonal evolution RNNs.
• Integrated a gating mechanism, incorporating reset and update gates, similar to Gated Recurrent Units (GRUs).
• Evaluated the model on benchmark tasks for long-term dependencies, including question answering and language modeling.

Main Results:

• The proposed RNN model significantly outperformed Long Short-Term Memory (LSTM), GRUs, and vanilla unitary/orthogonal RNNs.
• Empirical evidence showed that unitary and orthogonal RNNs inherently lack the ability to forget.
• Achieved competitive results on diverse natural sequential tasks and synthetic long-term dependency tasks.

Conclusions:

• The novel RNN model effectively balances remembering and forgetting, crucial for sequential data processing.
• The ability to forget is a vital component for RNN performance, often lacking in unitary/orthogonal variants.
• This new architecture offers a promising advancement for various natural language processing and time-series prediction applications.