Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Learning curves for stochastic gradient descent in linear feedforward networks.

Justin Werfel1, Xiaohui Xie, H Sebastian Seung

  • 1Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. jkwerfel@mit.edu

Neural Computation
|October 11, 2005
PubMed
Summary

Direct gradient descent and perturbation methods for linear perceptron training were analyzed. Node and weight perturbation methods showed slower learning speeds compared to direct gradient descent, especially in larger networks.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Uncovering Sex Differences in the Drosophila Ventral Nerve Cord Through Connectome Alignment.

bioRxiv : the preprint server for biology·2026
Same author

Eyewire II - A connectomic resource for resolving cell types and circuits of the mouse retina.

bioRxiv : the preprint server for biology·2026
Same author

Distributed control circuits across a brain-and-cord connectome.

Nature·2026
Same author

Near-inertial internal waves funneled into Taylor caps shape seamount tops, with implications for biodiversity in the deep ocean.

Science advances·2026
Same author

Beyond the cerebral cortex: cerebellar language-related subregions contributions to fluency in post-stroke aphasia.

NeuroImage. Clinical·2026
Same author

Case Report: Complete remission after radiotherapy in two cases of locally advanced extramammary Paget's disease of the perineum.

Frontiers in oncology·2026

Area of Science:

  • Machine Learning
  • Artificial Intelligence
  • Computational Neuroscience

Background:

  • Gradient-following learning methods are crucial for training artificial neural networks.
  • Implementation challenges with traditional gradient descent necessitate exploring stochastic variants.
  • Online training methods are vital for adaptive learning systems.

Purpose of the Study:

  • To analyze and compare the learning speeds of three online training methods for a linear perceptron.
  • To investigate the impact of perturbation methods (node and weight) versus direct gradient descent.
  • To characterize the influence of parameter choices and update uncertainties on learning curves.

Main Methods:

  • Comparative analysis of learning speeds for direct gradient descent, node perturbation, and weight perturbation.

Related Experiment Videos

  • Definition of learning speed as the rate of exponential decay in learning curves.
  • Optimization of scalar parameters controlling weight update size to maximize learning speed.

    • Main Results:

    • Node perturbation is slower than direct gradient descent by a factor equal to the number of output units.
    • Weight perturbation is slower than node perturbation by an additional factor equal to the number of input units.
    • Parallel perturbation offers faster learning than sequential perturbation, independent of network size.

    Conclusions:

    • Weight perturbation can be inefficient for large networks due to its significant slowdown.
    • Node perturbation's performance can rival direct gradient descent when output units are few.
    • The practical applicability of these methods is contingent on specific learning problem characteristics.