Evolution and generalization of a single neurone: I. Single-layer perceptron as seven statistical classifiers

Area of Science:

• Machine Learning
• Statistical Classification
• Artificial Neural Networks

Background:

• The behavior of non-linear single-layer perceptrons (SLPs) during training is often studied with fixed decision boundaries.
• Existing research typically does not explore the dynamic evolution of SLP decision boundaries in relation to diverse statistical classifiers.

Purpose of the Study:

• To investigate the non-linear single-layer perceptron (SLP) as a dynamic process with increasing weights and a gradually changing cost function.
• To demonstrate that SLP decision boundaries converge to those of seven specific statistical classifiers during backpropagation training.
• To introduce novel complexity-control techniques for expanding the range of classifiers achievable by SLPs.

Main Methods:

• Treating the non-linear single-layer perceptron (SLP) as a process with incrementally increasing weights and a sum of squares cost function.
• Utilizing backpropagation training to observe the evolution of the SLP's decision boundary.
• Proposing five new complexity-control techniques: target value control, data centering, zero weight initialization, anti-regularization, and an exponentially increasing learning step.

Main Results:

• The decision boundary of the SLP during backpropagation training becomes identical or very close to those of seven statistical classifiers: Euclidean distance, regularized linear discriminant analysis, standard Fisher linear discriminant, Fisher linear discriminant with pseudoinverse covariance, generalized Fisher discriminant, minimum empirical error, and maximum margin classifiers.
• The proposed complexity-control techniques enable the generation of a wider variety of classifiers from the SLP.

Conclusions:

• The non-linear single-layer perceptron, under specific training conditions, can effectively emulate a broad spectrum of statistical classifiers.
• The choice of classifier achieved by the SLP is contingent upon the dataset, cost function, optimization method, and stopping criteria.
• The introduction of novel complexity-control techniques enhances the versatility and applicability of SLPs in classification tasks.