Recurrent Neural Networks With External Addressable Long-Term and Working Memory for Learning Long-Term Dependences

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) struggle with learning long-term dependencies (LTDs) due to limited internal memory capacity.
• Existing neural external memory architectures face challenges in efficiently managing and accessing information for LTDs.

Purpose of the Study:

• To propose a novel external memory architecture for RNNs that enhances the learning of long-term dependencies.
• To introduce the external addressable long-term and working memory (EALWM)-augmented RNN architecture.
• To demonstrate the capability of the proposed architecture to learn LTDs without vanishing gradients.

Main Methods:

• Developed an EALWM-augmented RNN architecture.
• Implemented a distinct division of external memory into addressable long-term and working memory components.
• Conducted experiments on algorithm learning, language modeling, and question answering tasks.

Main Results:

• The EALWM-augmented RNN architecture demonstrated effective learning of long-term dependencies.
• The proposed model mitigated the vanishing gradient problem inherent in traditional RNNs for LTDs.
• Experimental results across diverse tasks indicate significant potential for practical applications.

Conclusions:

• The EALWM-augmented RNN offers a promising solution for overcoming limitations in learning long-term dependencies.
• The distinct and addressable memory components contribute to improved performance in complex sequence tasks.
• This architecture represents a significant advancement for RNNs in handling long-term information processing.