Learning temporal sequences from examples in a local feedback neural network

Statistical-mechanical techniques are used to study temporal sequence association in a local feedback neural network consisting of an input layer of context units and a single output unit. Each context unit has a feedback connection onto itself such that it accumulates an exponentially decaying moving average or trace of previous inputs to that unit. The formation of these traces allows the network to extract temporal information over an interval delta, where delta = 1/magnitude of ln gamma and gamma is the decay-rate of the moving average. The particular problem of learning a rule from examples is considered where the rule is generated by a teacher local feedback network. The replica method and other mean-field theory techniques are used to determine how the resulting generalization error of the network varies as a function of sequence length M and decay-rate gamma.