Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Weight groupings in second order training methods for recurrent networks.

L W Chan1, C C Szeto

  • 1Computer Science and Engineering Department, The Chinese University of Hong Kong, Shatin, Hong Kong. lwchan@cse.cuhk.edu.hk

International Journal of Neural Systems
|November 14, 2001
PubMed
Summary

This study introduces block-diagonal approximations for the Hessian matrix in Levenberg-Marquardt training of recurrent neural networks, significantly reducing training time without impacting generalization.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Management of chronic kidney disease: a Hong Kong consensus recommendation.

Hong Kong medical journal = Xianggang yi xue za zhi·2024
Same author

The Hong Kong Renal Registry: a recent update.

Hong Kong medical journal = Xianggang yi xue za zhi·2024
Same author

Home haemodialysis with a novel machine in a patient with end-stage kidney disease: first case report from Asia.

Hong Kong medical journal = Xianggang yi xue za zhi·2023
Same author

Impact of COVID-19 on the mortality of dialysis patients and kidney transplant recipients during the Omicron fifth wave in Hong Kong.

Hong Kong medical journal = Xianggang yi xue za zhi·2023
Same author

Hong Kong College of Physicians Position Statement and Recommendations on the 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension Guidelines for the Management of Arterial Hypertension.

Hong Kong medical journal = Xianggang yi xue za zhi·2020
Same author

Prenatal diagnosis of pathogenic genomic imbalance in fetuses with increased nuchal translucency but normal karyotyping using chromosomal microarray.

Hong Kong medical journal = Xianggang yi xue za zhi·2019

Area of Science:

  • Machine Learning
  • Artificial Intelligence
  • Neural Networks

Background:

  • Recurrent neural networks (RNNs) are powerful tools for sequential data.
  • Training RNNs can be computationally intensive, often employing optimization methods like Levenberg-Marquardt.
  • Approximating the Hessian matrix is a key strategy to improve training efficiency.

Purpose of the Study:

  • To investigate the efficacy of block-diagonal matrix approximations for the Hessian in RNN training.
  • To analyze the impact of different weight updating and grouping strategies on performance.
  • To compare these approximation methods against the standard Levenberg-Marquardt algorithm.

Main Methods:

  • Utilized block-diagonal matrix approximations to estimate the Hessian matrix.

Related Experiment Videos

  • Implemented and compared asynchronous and synchronous weight updating strategies.
  • Evaluated four distinct weight grouping methods: correlation, k-unit, layer, and arbitrary blocks.
  • Analyzed computational complexity, approximation accuracy, and training duration.

    • Main Results:

    • Block-diagonal approximation methods substantially reduced RNN training time.
    • These methods demonstrated comparable or improved generalization ability compared to the original Levenberg-Marquardt method.
    • Different weight grouping and updating strategies showed varying impacts on efficiency and accuracy.

    Conclusions:

    • Block-diagonal Hessian approximations offer a computationally efficient alternative for training RNNs.
    • The choice of weight grouping and updating strategy can be optimized for specific network architectures and tasks.
    • This approach provides a practical method for accelerating deep learning model training.