Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Fast curvature matrix-vector products for second-order gradient descent.

Nicol N Schraudolph1

  • 1IDSIA, Galleria 2, 6928 Manno, Switzerland. nic@inf.ethz.ch

Neural Computation
|June 25, 2002
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

A role for codon order in translation dynamics.

Cell·2010
Same author

Graph kernels for disease outcome prediction from protein-protein interaction networks.

Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing·2007
Same author

Gradient-based manipulation of nonparametric entropy estimates.

IEEE transactions on neural networks·2004
Same journal

A Model-Free Reinforcement Learning Implementation of Decision Making Under Uncertainty by Sequential Sampling.

Neural computation·2026
Same journal

DROP: Distributional and Regular Optimism and Pessimism for Reinforcement Learning.

Neural computation·2026
Same journal

Hierarchical Active Inference Using Successor Representations.

Neural computation·2026
Same journal

W-Kernel and Its Principal Space for Frequentist Evaluation of Bayesian Estimators.

Neural computation·2026
Same journal

A Hidden Markov Model-Inspired Sequence Classification Method for Hyperdimensional Computing.

Neural computation·2026
Same journal

Sparse Graphical Modeling for Electrophysiological Phase-Based Connectivity Using Circular Statistics.

Neural computation·2026
See all related articles

We present a novel method for approximating second-order gradient steps efficiently. This approach offers new insights and improvements for on-line learning techniques like stochastic meta-descent (SMD).

Area of Science:

  • Machine Learning
  • Optimization Algorithms
  • Numerical Analysis

Background:

  • Second-order gradient methods (Newton, Gauss-Newton, Levenberg-Marquardt, natural gradient) are crucial for optimization but computationally intensive.
  • Existing acceleration techniques for on-line learning, such as matrix momentum and stochastic meta-descent (SMD), have been developed through distinct theoretical frameworks.

Purpose of the Study:

  • To introduce a generic, computationally efficient method for approximating various second-order gradient steps.
  • To provide a unified perspective on existing acceleration techniques and derive potential improvements.

Main Methods:

  • Developing a method to approximate second-order gradient steps using specialized curvature matrix-vector products.
  • Achieving linear time complexity per iteration for these approximations, with matrix-vector products computable in O(n).

Related Experiment Videos

  • Demonstrating that matrix momentum and stochastic meta-descent (SMD) are specific instances of this generic approach.

    • Main Results:

    • The proposed method enables efficient iterative approximation of Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient steps.
    • The computational cost per iteration is reduced to linear time.
    • The unified framework provides novel insights into the operation of matrix momentum and SMD, leading to enhanced performance for SMD.

    Conclusions:

    • A generic and efficient method for approximating second-order gradient steps has been successfully proposed.
    • This work unifies and improves upon existing on-line learning acceleration techniques, particularly SMD.
    • The findings offer a new perspective on the interplay between optimization algorithms and acceleration methods in machine learning.