Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Videos

Learning long-term dependencies with gradient descent is difficult.

Y Bengio¹, P Simard, P Frasconi

¹Dept. d'Inf. et de Recherche Oper., Montreal Univ., Que.

IEEE Transactions on Neural Networks

|January 1, 1994

Summary

This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Universal autofocus for quantitative volumetric microscopy of whole mouse brains.

Nature methods·2021

Same author

Temporal and spatial mapping of red grouper Epinephelus morio sound production.

Journal of fish biology·2014

Same author

Visual complaints from Hong Kong repatriates.

Treatment services bulletin. Canada. Department of Veterans' Affairs·2010

Same author

Adaptive importance sampling to accelerate training of a neural probabilistic language model.

IEEE transactions on neural networks·2008

Same author

A general framework for adaptive processing of data structures.

IEEE transactions on neural networks·2008

Same author

Taking on the curse of dimensionality in joint distributions using neural networks.

IEEE transactions on neural networks·2008

Same journal

Universal perceptron and DNA-like learning algorithm for binary neural networks: LSBF and PBF implementations.

IEEE transactions on neural networks·2013

Same journal

Guest editorial: special section on white box nonlinear prediction models.

IEEE transactions on neural networks·2011

Same journal

Data-based fault-tolerant control of high-speed trains with traction/braking notch nonlinearities and actuator failures.

IEEE transactions on neural networks·2011

Same journal

Guest editorial: special section on data-based control, modeling, and optimization.

IEEE transactions on neural networks·2011

Same journal

Neural network-based multiple robot simultaneous localization and mapping.

IEEE transactions on neural networks·2011

Same journal

Data-driven model-free adaptive control for a class of MIMO nonlinear discrete-time systems.

IEEE transactions on neural networks·2011

See all related articles

Training recurrent neural networks for long-term dependencies is challenging for gradient descent algorithms. This study explains why and explores alternative learning methods for improved sequence modeling.

Area of Science:

Artificial Intelligence
Machine Learning
Deep Learning

Background:

Recurrent neural networks (RNNs) are effective for sequence-to-sequence tasks like prediction and recognition.
Training RNNs on tasks with long temporal dependencies presents significant practical challenges.

Purpose of the Study:

To elucidate the difficulties gradient-based learning algorithms encounter when capturing long-range dependencies in sequences.
To identify the inherent trade-off between efficient gradient descent learning and maintaining information over extended periods.

Main Methods:

Theoretical analysis of gradient-based learning algorithms in the context of sequential data.
Investigation into the mathematical underpinnings of vanishing or exploding gradients with increasing sequence length.

Related Experiment Videos

Main Results:

Gradient-based learning algorithms face exponentially increasing difficulty as the length of temporal dependencies grows.
A fundamental trade-off exists between the efficiency of gradient descent and the network's ability to retain information over long intervals.

Conclusions:

The limitations of standard gradient descent for long-interval temporal dependencies are clearly demonstrated.
Understanding this trade-off necessitates the exploration and development of alternative training methodologies for RNNs.