Related Experiment Video
Updated: Jan 11, 2026
13:19
Deep Neural Networks for Image-Based Dietary Assessment
Published on: March 13, 2021
9.9K
Effective Learning Rules as Natural Gradient Descent
Lucas Shoji1, Kenta Suzuki2, Leo Kozachkov3,4
1Department of Physics and Department of Brain and Cognitive Sciences, MIT, Cambridge, MA 02139, USA lshoji@mit.edu.
Neural Computation
|November 14, 2025
Summary
Effective learning rules can be unified as natural gradient descent. This finding reveals the gradient as a fundamental element across diverse learning processes, with broad implications for AI and neuroscience.
Area of Science:
- Machine Learning
- Computational Neuroscience
- Control Theory
Background:
- Learning rules aim to optimize performance over time.
- Current learning rules are diverse and often specialized.
- A unified mathematical framework for learning is lacking.
Purpose of the Study:
- To establish a general mathematical framework for effective learning rules.
- To demonstrate that a broad class of learning rules are instances of natural gradient descent.
- To unify understanding of gradient-based optimization in learning.
Main Methods:
- Expressing learning rules as natural gradient descent.
- Defining appropriate information metrics for different learning settings.
- Analyzing parameter updates using matrix calculus.
Main Results:
- A broad class of effective learning rules are shown to be natural gradient descent.
- Parameter updates are demonstrated to be a product of a positive-definite matrix and a loss gradient.
- The framework applies to continuous-time, discrete-time, stochastic, and higher-order learning rules.
Conclusions:
- The gradient is a fundamental object underlying all learning processes.
- This work provides a unified theoretical framework for understanding learning.
- The findings have practical implications for artificial intelligence, control systems, and experimental neuroscience.
Related Concept Videos
01:14Gradient and Del Operator
4.3K
In mathematics and physics, the gradient and del operator are fundamental concepts used to describe the behavior of functions and fields in space. The gradient is a mathematical operator that gives both the magnitude and direction of the maximum spatial rate of change. Consider a person standing on a mountain. The slope of the mountain at any given point is not defined unless it is quantified in a particular direction. For this reason, a "directional derivative" is defined, which is a vector...
4.3K
01:52Regression Toward the Mean
6.8K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.8K
01:12Observational Learning
804
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
804
01:18Introduction to Learning
912
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
912
01:22Purposive Learning
428
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
428
01:32What is Natural Selection?
125.6K
Natural selection is an evolutionary process in which individuals with survival-promoting traits reproduce at higher rates. These favorable traits become more common within a population or species. Naturally selected traits initially arise via random genetic mutations. In order for selection to occur, there must be variation within a population, the trait controlling the variation must be heritable, and there must be an evolutionary advantage for variation in the trait.
125.6K