Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies.

Tsungnan Lin1, Bill G. Horne, C Lee Giles

  • 1EPSON Palo Alto Laboratory, Palo Alto, USA

Neural Networks : the Official Journal of the International Neural Network Society
|March 29, 2003
PubMed
Summary

Increasing the embedded memory order in recurrent neural networks (RNNs) significantly improves their ability to learn long-term temporal dependencies across various architectures. This enhancement makes RNNs more robust for complex sequence learning tasks.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Automating document classification with distant supervision to increase the efficiency of systematic reviews: A case study on identifying studies with HIV impacts on female sex workers.

PloS one·2022
Same author

Understanding the onset of hot streaks across artistic, cultural, and scientific careers.

Nature communications·2021
Same author

<i>Screenomics</i>: A Framework to Capture and Analyze Personal Life Experiences and the Ways that Technology Shapes Them.

Human-computer interaction·2021
Same author

Large Scale Subject Category Classification of Scholarly Papers With Deep Attentive Neural Networks.

Frontiers in research metrics and analytics·2021
Same author

An Entropy Metric for Regular Grammar Classification and Learning with Recurrent Neural Networks.

Entropy (Basel, Switzerland)·2021
Same author

Farm-to-table: A Situation Awareness Model for Food Safety Assurance for Porous Borders.

Comprehensive reviews in food science and food safety·2021

Area of Science:

  • Artificial Intelligence
  • Machine Learning
  • Deep Learning

Background:

  • Recurrent Neural Networks (RNNs) struggle with learning long-term temporal dependencies.
  • NARX (Nonlinear Autoregressive with Exogenous Inputs) networks show improved performance due to 'jump-ahead' connections facilitating gradient propagation.
  • The hypothesis is that increased memory order, not just architecture, enhances long-term dependency learning.

Purpose of the Study:

  • To empirically validate if increasing embedded memory order improves long-term dependency learning in RNNs.
  • To investigate this effect across different RNN architectures: globally recurrent, locally recurrent, and NARX networks.
  • To provide practical insights for enhancing RNN robustness in sequence learning.

Main Methods:

Related Experiment Videos

  • Comparative analysis of three RNN architectures (globally recurrent, locally recurrent, NARX).
  • Systematic variation of embedded memory orders within each architecture.
  • Evaluation on two distinct simple long-term dependency problems.

    • Main Results:

    • All tested RNN architectures demonstrated significant improvements in learning long-term dependencies as embedded memory order increased.
    • Increased memory order consistently enhanced performance across globally recurrent, locally recurrent, and NARX networks.
    • The findings support the hypothesis that memory order is a key factor for robust long-term dependency learning.

    Conclusions:

    • Increasing the embedded memory order is an effective strategy to improve RNN performance on long-term dependency tasks.
    • This approach offers a general method to enhance the robustness of various RNN architectures without fundamental architectural changes.
    • Users can improve their existing RNN models' capability for sequence learning by adjusting the memory order.