A dynamical model for the analysis and acceleration of learning in feedforward networks

A dynamical system model is derived for feedforward neural networks with one layer of hidden nodes. The model is valid in the vicinity of flat minima of the cost function that rise due to the formation of clusters of redundant hidden nodes with nearly identical outputs. The derivation is carried out for networks with an arbitrary number of hidden and output nodes and is, therefore, a generalization of previous work valid for networks with only two hidden nodes and one output node. The Jacobian matrix of the system is obtained, whose eigenvalues characterize the evolution of learning. Flat minima correspond to critical points of the phase plane trajectories and the bifurcation of the eigenvalues signifies their abandonment. Following the derivation of the dynamical model, we show that identification of the hidden nodes clusters using unsupervised learning techniques enables the application of a constrained application (Dynamically Constrained Back Propagation-DCBP) whose purpose is to facilitate prompt bifurcation of the eigenvalues of the Jacobian matrix and, thus, accelerate learning. DCBP is applied to standard benchmark tasks either autonomously or as an aid to other standard learning algorithms in the vicinity of flat minima. Its application leads to significant reduction in the number of required epochs for convergence.