Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

The general inefficiency of batch training for gradient descent learning.

D Randall Wilson1, Tony R Martinez

  • 1Fonix Corporation, 180 West Election Road Suite 200, Draper, UT, USA. randy@axon.cs.byu.edu

Neural Networks : the Official Journal of the International Neural Network Society
|November 19, 2003
PubMed
Summary

On-line training for neural networks is significantly faster than batch training, often by orders of magnitude. This speed advantage stems from on-line training

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same journal

Injecting reasoning into vision-language models via weight-decomposed merging.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

A Non-equilibrium thermodynamic framework for neural networks: A principled correspondence and parameter dynamics.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Dual-pathway mask ranking guided selective fine-tuning for backdoor purification.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Discriminative transfer feature learning for unsupervised domain adaptation.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

ARetinex-Net: Low-light image enhancement with adaptive retinex model and global illumination representation.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

A Multi-Task Learning Framework with Physics Embedded for Signal Reconstruction and State Prediction in Underground Multi-Robot Systems.

Neural networks : the official journal of the International Neural Network Society·2026

Area of Science:

  • Machine Learning
  • Artificial Intelligence
  • Neural Networks

Background:

  • Neural network training utilizes gradient descent, with two primary methods: batch and on-line training.
  • A common misconception suggests batch training is superior due to a more accurate gradient approximation.

Purpose of the Study:

  • To debunk the myth that batch training is faster or more accurate than on-line training.
  • To elucidate the reasons behind the performance differences between batch and on-line gradient descent.

Main Methods:

  • Comparative analysis of batch versus on-line gradient descent algorithms.
  • Empirical evaluation on a large-scale speech recognition task (20,000 instances) and 26 additional learning tasks.

Main Results:

Related Experiment Videos

  • Batch training is consistently slower than on-line training, particularly with large datasets.
  • On-line training allows for larger learning rates by adapting to error surface curves, leading to faster convergence.
  • No significant difference in accuracy was observed between the two training methods.

Conclusions:

  • On-line training offers a substantial speed advantage over batch training for neural networks.
  • The perceived benefits of batch training's gradient approximation are outweighed by its slower convergence rates.
  • On-line training is a more efficient method for training neural networks, especially on large datasets.