Dynamics of learning near singularities in radial basis function networks
1Amari Research Unit, RIKEN Brain Science Institute, Saitama, 3510198, Japan. weihaikun@brain.riken.jp
Summary
Radial basis function (RBF) networks exhibit plateau phenomena during learning due to singularities. This study analyzes these dynamical behaviors near overlap and elimination singularities in RBF networks.
Area of Science:
- Machine Learning
- Artificial Intelligence
- Computational Neuroscience
Background:
- Radial basis function (RBF) networks are widely used for function approximation in regression.
- Learning in RBF networks involves iterative refinement based on observed data.
- Difficulties arise when basis functions become identical or their magnitudes approach null, leading to component reduction.
Purpose of the Study:
- To analyze the dynamical behaviors of learning near overlap and elimination singularities in RBF networks.
- To investigate the causes and characteristics of plateau phenomena during RBF network training.
- To provide a detailed dynamical analysis applicable to both on-line and batch learning modes.
Main Methods:
- Analysis of averaged learning equations for RBF networks.
- Explicit eigenvalue calculation of the Hessian matrix to assess stability at overlap singularities.
- Plotting analytical dynamic vector fields near singularities.
- Comparison of analytical fields with real trajectories from numerical methods.
- Simulation to confirm plateau phenomena in both batch and on-line learning.
Main Results:
- Identified overlap and elimination singularities as sources of unusual learning dynamics.
- Demonstrated that component reduction occurs at singular regions.
- Confirmed the existence of plateau phenomena in both batch and on-line learning simulations.
- Provided analytical insights into the stability at overlap singularities.
Conclusions:
- Singularities in RBF networks lead to component reduction and plateau phenomena.
- The study offers a detailed dynamical analysis of these behaviors near singularities.
- Findings are applicable to both on-line and batch learning paradigms, enhancing understanding of RBF network training.
Related Concept Videos
01:26Linear Approximation in Frequency Domain
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear.
01:24Pole and System Stability
The transfer function is a fundamental concept representing the ratio of two polynomials. The numerator and denominator encapsulate the system's dynamics. The zeros and poles of this transfer function are critical in determining the system's behavior and stability.
Simple poles are unique roots of the denominator polynomial. Each simple pole corresponds to a distinct solution to the system's characteristic equation, typically resulting in exponential decay terms in the system's response.
Simple poles are unique roots of the denominator polynomial. Each simple pole corresponds to a distinct solution to the system's characteristic equation, typically resulting in exponential decay terms in the system's response.
01:30Current Growth And Decay In RL Circuits
The current growth and decay in RL circuits can be understood by considering a series RL circuit consisting of a resistor, an inductor, a constant source of emf, and two switches. When the first switch is closed, the circuit is equivalent to a single-loop circuit consisting of a resistor and an inductor connected to a source of emf. In this case, the source of emf produces a current in the circuit. If there were no self-inductance in the circuit, the current would rise immediately to a steady...
01:23Root Loci for Positive-Feedback Systems
The Hartley oscillator is a positive feedback system that sustains oscillations by feeding the output back to the input in phase, thereby reinforcing the signal. Positive feedback systems can be viewed as negative feedback systems with inverted feedback signals. In these systems, the root locus encompasses all points on the s-plane where the angle of the system transfer function equals 360 degrees.
The construction rules for the root locus in positive feedback systems are similar to those in...
The construction rules for the root locus in positive feedback systems are similar to those in...
01:29Second Derivatives and the Shape of a Graph
The second derivative of a function provides essential information about a graph's curvature and how it changes over an interval. It helps determine whether a function is concave upward or concave downward and identifies points where the curvature changes. These properties are fundamental in analyzing real-world scenarios, such as changes in road elevation, population growth, and economic trends.A function f(x) is considered concave upward on an interval if its graph lies above all its tangent...
01:25Modeling with Differential Equations
Population dynamics can be described mathematically by considering the population size P(t) as a function of time. The rate of change of the population is then represented by the derivative of P(t). A simple assumption is that the rate of growth is proportional to the size of the population itself. This leads to an exponential growth model, where the population increases rapidly without bound. While this is a useful first approximation, it does not reflect realistic long-term...