Neural network approximation: Three hidden layers are enough.

Area of Science:

• Computational mathematics
• Machine learning theory
• Neural network approximation

Background:

• Traditional neural networks face challenges with the curse of dimensionality in function approximation.
• The need for neural network architectures with enhanced approximation capabilities is critical for complex problems.
• Understanding the theoretical limits and potential of different activation functions is an ongoing research area.

Purpose of the Study:

• To introduce a novel neural network architecture, Floor-Exponential-Step (FLES) networks, with enhanced approximation power.
• To analyze the approximation rates of FLES networks for Hölder continuous and general continuous functions.
• To demonstrate the ability of FLES networks to overcome the curse of dimensionality.

Main Methods:

• Development of a three-hidden-layer neural network utilizing floor, exponential, and step functions as activation functions.
• Mathematical analysis of uniform approximation rates for Hölder continuous functions on [0,1]^d.
• Extension of approximation results to general continuous functions and bounded domains, including L^p norms.

Main Results:

• FLES networks with width max{d,N} and three hidden layers achieve an exponential approximation rate of 3λ(2d)^α 2^{-αN} for Hölder continuous functions.
• For general continuous functions, the constructive approximation rate is 2ω_f(2d)2^{-N} + ω_f(2d2^{-N}).
• The proposed networks mitigate the curse of dimensionality, with approximation rates largely independent of the input dimension 'd'.

Conclusions:

• Floor-Exponential-Step networks represent a significant advancement in neural network approximation theory.
• These networks offer a powerful tool for approximating complex functions while avoiding the curse of dimensionality.
• The findings have implications for deep learning theory and the design of efficient neural network architectures.