Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

On 'natural' learning and pruning in multi-layered perceptrons.

T Heskes1

  • 1Foundation for Neural Networks, University of Nijmegen, 6525 EZ Nijmegen, The Netherlands.

Neural Computation
|April 19, 2000
PubMed
Summary

Natural gradient descent offers superior efficiency for online learning compared to standard methods. This research explores its application in batch learning and pruning, introducing faster algorithms using a Fisher matrix approximation for multilayered perceptrons.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Role of conduct problems in the relation between Attention-Deficit Hyperactivity disorder, substance use, and gaming.

European neuropsychopharmacology : the journal of the European College of Neuropsychopharmacology·2018
Same author

Self-organizing maps, vector quantization, and mixture modeling.

IEEE transactions on neural networks·2008
Same author

Prediction of bladder outlet obstruction in men with lower urinary tract symptoms using artificial neural networks.

The Journal of urology·1999
Same author

Partial retraining: a new approach to input relevance determination.

International journal of neural systems·1999
Same author

Pruning using parameter and neuronal metrics.

Neural computation·1999
Same author

How dependencies between successive examples affect on-line learning.

Neural computation·1996

Area of Science:

  • Machine Learning
  • Computational Neuroscience
  • Optimization Theory

Background:

  • Standard gradient descent is less efficient for online learning.
  • Natural gradient descent (NGD) shows greater efficiency in prior studies.
  • The Fisher matrix is crucial for NGD and related optimization algorithms.

Purpose of the Study:

  • To re-derive natural gradients and explore their implications for batch learning and pruning.
  • To connect natural gradients to existing algorithms like Levenberg-Marquardt and Optimal Brain Surgeon.
  • To develop faster and more robust learning and pruning algorithms for multilayered perceptrons.

Main Methods:

  • Derivation of natural gradients using a novel approach.
  • Analysis of implications for batch-mode learning and network pruning.

Related Experiment Videos

  • Development of a layered approximation for the Fisher matrix in multilayered perceptrons.
  • Comparison of performance using the approximated Fisher matrix versus the exact Fisher matrix.

    • Main Results:

    • Natural gradients offer advantages in efficiency for both online and batch learning.
    • The Fisher matrix approximation leads to significantly faster natural learning algorithms.
    • The proposed methods enhance the robustness of pruning procedures in neural networks.
    • Connections were established between natural gradients, Fisher matrix, and algorithms like Levenberg-Marquardt and Optimal Brain Surgeon.

    Conclusions:

    • A refined understanding of natural gradients enhances their applicability to broader machine learning tasks.
    • Layered Fisher matrix approximation provides a computationally efficient alternative for multilayered perceptrons.
    • The developed natural learning algorithms and pruning procedures demonstrate improved speed and robustness.