A dynamical model for the analysis and acceleration of learning in feedforward networks

Area of Science:

• Computational Neuroscience
• Machine Learning
• Dynamical Systems Theory

Background:

• Feedforward neural networks can develop clusters of redundant hidden nodes, leading to flat minima in the cost function.
• These flat minima complicate the learning process by slowing down convergence.

Purpose of the Study:

• To derive a generalized dynamical system model for feedforward neural networks with one hidden layer.
• To introduce a method for accelerating learning in the presence of flat minima.

Main Methods:

• Derivation of a dynamical system model for feedforward neural networks, valid near flat minima.
• Analysis of the Jacobian matrix eigenvalues to characterize learning dynamics.
• Application of unsupervised learning to identify hidden node clusters.
• Development and application of Dynamically Constrained Back Propagation (DCBP).

Main Results:

• The derived model generalizes previous work and characterizes learning via Jacobian eigenvalues.
• Flat minima are identified as critical points where eigenvalue bifurcation signifies abandonment.
• Dynamically Constrained Back Propagation (DCBP) effectively facilitates eigenvalue bifurcation.
• DCBP application significantly reduces the number of epochs required for convergence on benchmark tasks.

Conclusions:

• The dynamical system model provides insights into learning dynamics near flat minima.
• Dynamically Constrained Back Propagation (DCBP) is an effective technique for accelerating neural network training.
• DCBP can be used autonomously or in conjunction with other algorithms to improve learning efficiency.