Learning algorithms based on linearization

The aim of this article is to investigate a mechanical description of learning. A framework for local and simple learning algorithms based on interpreting a neural network as a set of configuration constraints is proposed. For any architectural design and learning task, unsupervised and supervised algorithms can be derived, optionally using unconstrained and hidden neurons. Unlike algorithms based on the gradient in weight space, the proposed tangential correlation (TC) algorithms move along the gradient in state space. This results in optimal scaling properties and simple expressions for the weight updates. The number of synapses is much larger than the number of neurons. A constraint for neural states does not impose a unique constraint for synaptic weights. Which weights to assign credit to can be selected from a parametrization of all weight changes equivalently satisfying the state constraints. At the heart of the parametrization are minimal weight changes. Two supervised algorithms (differing by their parametrizations) operating on a three-layer perceptron are compared with standard backpropagation. The successful training of fixed points of recurrent networks is demonstrated. The unsupervised learning of oscillations with variable frequencies is performed on standard and more sophisticated recurrent networks. The results presented here can be useful both for the analysis and for the synthesis of learning algorithms.