On the training and performance of high-order neural networks

An extensive study was undertaken on the architecture, training, and properties of neural networks of order higher than 1. The formulation of the training of high-order neural networks as a nonlinear associative recall problem provides the basis for their optimal least squares training. The simplicity of the outer-product rule motivates the study of the approximation of optimal least squares training by the outer-product rule and the effect of the network order on the efficiency of this approximation. Neural networks with composite key patterns are subsequently proposed as the natural generalization of neural networks of order higher than 1. The properties of this class of neural networks are revealed by studying their optimal least squares training and its relationship with the outer-product rule. The performance of neural networks with composite key patterns is analytically evaluated by studying their capacity. The properties and performance bounds provided by the analytical study are verified through experimental results.