Wide and deep neural networks achieve consistency for classification

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Neural networks are widely used for classification, but their consistency for arbitrary data distributions remains an open problem.
• Standard training procedures do not guarantee that neural networks minimize misclassification probability.

Purpose of the Study:

• To identify and construct explicit neural network classifiers that are consistent.
• To analyze the properties of infinitely wide and deep neural networks for classification.

Main Methods:

• Analysis of infinitely wide and deep neural networks using the connection to neural tangent kernels.
• Identification of explicit activation functions that lead to consistent classifiers.
• Development of a taxonomy for infinitely wide and deep networks.

Main Results:

• Explicit activation functions were identified that construct consistent neural network classifiers.
• These networks, depending on the activation function, implement 1-nearest neighbor, majority vote, or singular kernel classifiers.
• The identified activation functions are simple and easy to implement, differing from common ones like ReLU or sigmoid.

Conclusions:

• Deep neural networks offer benefits for classification tasks, unlike regression tasks where excessive depth can be detrimental.
• The study provides a theoretical foundation for constructing consistent neural network classifiers.
• Novel activation functions are proposed for achieving classification consistency in deep learning models.