Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Regularization parameter estimation for feedforward neural networks.

Ping Guo1, M R Lyu, C P Chen

  • 1Dept. of Comput. Sci., Beijing Normal Univ., China.

IEEE Transactions on Systems, Man, and Cybernetics. Part B, Cybernetics : a Publication of the IEEE Systems, Man, and Cybernetics Society
|February 2, 2008
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Modifiable factors related to 7-year renal outcomes in subjects with type 2 diabetes and chronic kidney disease stage 3.

Journal of postgraduate medicine·2020
Same author

Re-focusing on Agnathia-Otocephaly complex.

Clinical oral investigations·2020
Same author

YAP/TAZ Regulate Elevation and Bone Formation of the Mouse Secondary Palate.

Journal of dental research·2020
Same author

First trimester placental vascular indices and volume by three-dimensional ultrasound in pre-gravid overweight women.

Placenta·2019
Same author

[Study on the mechanism of serotype conversion of historical isolates of <i>Shigella flexneri</i>].

Zhonghua yu fang yi xue za zhi [Chinese journal of preventive medicine]·2018
Same author

Axillary web syndrome following epidermal inclusion cyst: a case report and literature review.

Clinical and experimental dermatology·2018
Same journal

Strategic Ability Updating in Concurrent Games by Coalitional Commitment.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2015
Same journal

Meta-Analysis of the First Facial Expression Recognition Challenge.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2012
Same journal

Adjustable model-based fusion method for multispectral and panchromatic images.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2012
Same journal

Face Feature Weighted Fusion Based on Fuzzy Membership Degree for Video Face Recognition.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2012
Same journal

A New Adaptive Fast Cellular Automaton Neighborhood Detection and Rule Identification Algorithm.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2012
Same journal

Human-arm-and-hand-dynamic model with variability analyses for a stylus-based haptic interface.

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society·2012
See all related articles

This study connects Gaussian probability functions in feedforward neural networks (NNs) to Tikhonov regularization using Kullback-Leibler distance. An estimation formula for regularization parameters is derived and validated in small, sparse datasets.

Area of Science:

  • Machine Learning
  • Computational Statistics
  • Neural Network Theory

Background:

  • Feedforward neural networks (NNs) often require regularization to prevent overfitting.
  • The Kullback-Leibler (KL) divergence is a fundamental measure of difference between probability distributions.
  • Kernel density estimation (KDE) is a non-parametric way to estimate probability density functions.

Purpose of the Study:

  • To establish a theoretical link between Gaussian probability functions in NNs and Tikhonov regularization.
  • To investigate the role of the smoothing parameter in KDE as a regularization parameter.
  • To derive and validate a formula for estimating regularization parameters from training data.

Main Methods:

  • Utilizing the Kullback-Leibler (KL) distance framework.

Related Experiment Videos

  • Analyzing a specific case of Gaussian probability functions within feedforward neural networks (NNs).
  • Deriving an estimation formula for regularization parameters through mathematical approximations.

    • Main Results:

    • A particular Gaussian probability function for NNs was shown to reduce to a first-order Tikhonov regularizer.
    • The smoothing parameter in kernel density estimation was identified as analogous to the regularization parameter.
    • An effective estimation formula for regularization parameters was derived and validated.

    Conclusions:

    • The derived estimation formula for regularization parameters performs well, especially in scenarios with sparse or small training datasets.
    • This work provides a theoretical bridge between probabilistic modeling in NNs and regularization techniques.
    • The findings offer practical implications for optimizing NN training with limited data.