Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Videos

Gradient-based manipulation of nonparametric entropy estimates.

Nicol N Schraudolph¹

¹nic@schraudolph.org

IEEE Transactions on Neural Networks

|October 6, 2004

Summary

This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

A role for codon order in translation dynamics.

Cell·2010

Same author

Graph kernels for disease outcome prediction from protein-protein interaction networks.

Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing·2007

Same author

Fast curvature matrix-vector products for second-order gradient descent.

Neural computation·2002

Same journal

Universal perceptron and DNA-like learning algorithm for binary neural networks: LSBF and PBF implementations.

IEEE transactions on neural networks·2013

Same journal

Guest editorial: special section on white box nonlinear prediction models.

IEEE transactions on neural networks·2011

Same journal

Data-based fault-tolerant control of high-speed trains with traction/braking notch nonlinearities and actuator failures.

IEEE transactions on neural networks·2011

Same journal

Guest editorial: special section on data-based control, modeling, and optimization.

IEEE transactions on neural networks·2011

Same journal

Neural network-based multiple robot simultaneous localization and mapping.

IEEE transactions on neural networks·2011

Same journal

Data-driven model-free adaptive control for a class of MIMO nonlinear discrete-time systems.

IEEE transactions on neural networks·2011

See all related articles

This study introduces novel nonparametric learning rules to optimize Shannon entropy for adaptive systems using kernel density estimation. These efficient rules handle data limitations and enable gradient-based optimization of output density shape.

Area of Science:

Machine Learning
Information Theory
Adaptive Systems

Background:

Traditional entropy optimization often relies on parametric assumptions about data distribution.
Nonparametric methods offer flexibility but can face challenges with quantized data and limited sample sizes.
Gradient-based optimization requires smooth and differentiable objective functions.

Purpose of the Study:

To derive differential learning rules for optimizing Shannon entropy in adaptive systems.
To develop a nonparametric approach that avoids assumptions on output density functional form.
To address practical issues like quantized data and finite sample sizes.

Main Methods:

Kernel density estimation (KDE) for nonparametric probability density function estimation.

Related Experiment Videos

Maximum likelihood techniques for regularizer optimization.

Development of a normalized entropy estimate invariant to affine transformations.

Application of gradient descent for optimizing the derived entropy estimates.

Main Results:

A family of simple and efficient differential learning rules derived from optimizing Shannon entropy.
The learning rules operate on pairs of input samples, suitable for various data constraints.
The normalized entropy estimate facilitates optimization of output density shape.
The method is amenable to fully online implementations.

Conclusions:

The proposed nonparametric approach provides effective and flexible entropy optimization for adaptive systems.
The derived learning rules are computationally efficient and adaptable to different scenarios.
This work offers a robust framework for adaptive systems requiring density estimation and entropy maximization.