Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Flat minima

S Hochreiter1, J Schmidhuber

  • 1Fakultät für Informatik, Technische Universität München, Germany.

Neural Computation
|January 1, 1997
PubMed
Summary
This summary is machine-generated.

We developed a novel algorithm to find simple neural networks that generalize well by searching for flat minima in error functions. This approach effectively prunes network components and outperforms other methods in stock market prediction.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

African swine fever: a global factor affecting agricultural markets over the medium term.

Revue scientifique et technique (International Office of Epizootics)·2022
Same author

Optimising traceability in trade for live animals and animal products with digital technologies.

Revue scientifique et technique (International Office of Epizootics)·2020
Same author

Salvage therapy with everolimus reduces the severity of treatment-refractory chronic GVHD without impairing disease control: a dual center retrospective analysis.

Bone marrow transplantation·2014
Same author

Introductory lecture the epidemiology and determinants of obesity in developed and developing countries.

International journal for vitamin and nutrition research. Internationale Zeitschrift fur Vitamin- und Ernahrungsforschung. Journal international de vitaminologie et de nutrition·2007
Same author

Learning nonregular languages: a comparison of simple recurrent networks and LSTM.

Neural computation·2002
Same author

Learning to forget: continual prediction with LSTM.

Neural computation·2000

Area of Science:

  • Machine Learning
  • Computational Neuroscience
  • Artificial Intelligence

Background:

  • Neural networks often suffer from overfitting, limiting their generalization capability.
  • Existing methods for improving generalization may rely on restrictive assumptions like Gaussian distributions or specific weight priors.

Purpose of the Study:

  • To introduce a new algorithm for identifying low-complexity neural networks with enhanced generalization.
  • To leverage the concept of "flat" minima in error functions as an indicator of network simplicity and reduced overfitting.

Main Methods:

  • The algorithm searches for flat minima, defined as large regions in weight space with constant error.
  • It employs a Minimum Description Length (MDL)-based Bayesian argument and a novel generalization error decomposition.

Related Experiment Videos

  • The method utilizes a prior over input-output functions, considering network architecture and training data, and requires second-order derivatives.

    • Main Results:

    • The algorithm automatically prunes network units, weights, and input connections.
    • Experiments with feedforward and recurrent networks demonstrate its effectiveness.
    • In stock market prediction, this flat minimum search significantly outperformed standard backpropagation, weight decay, and Optimal Brain Damage/Surgeon.

    Conclusions:

    • Flat minima in error functions are indicative of simple neural networks with low expected overfitting.
    • The proposed algorithm offers an effective, assumption-light approach to improving neural network generalization.
    • This method shows practical advantages in complex prediction tasks like financial market analysis.