Simultaneous neural network approximation for smooth functions.

Area of Science:

• Numerical Analysis
• Machine Learning
• Scientific Computing

Background:

• Deep neural networks (DNNs) are increasingly used for solving complex mathematical problems.
• Numerical solvers for partial differential equations (PDEs) are a key application area for DNNs.
• Understanding the approximation capabilities of DNNs in Sobolev norms is essential for their reliable application.

Purpose of the Study:

• To establish nonasymptotic approximation results for DNNs approximating smooth functions in Sobolev norms.
• To provide explicit error bounds dependent on network width and depth.
• To inform the development of DNN-based numerical solvers for PDEs.

Main Methods:

• Theoretical analysis of deep ReLU networks and their variants (using ReLU or ReLU squared activation functions).
• Derivation of nonasymptotic approximation error bounds for functions in Sobolev spaces (W^n,p).
• Characterization of error bounds in terms of network width (N) and depth (L).

Main Results:

• Deep ReLU networks with width O(NlogN) and depth O(LlogL) achieve approximation rates of O(N^(-2(s-1)/d)L^(-2(s-1)/d)) in the W^1,p norm for functions in C^s.
• Networks using ReLU or ReLU squared activations achieve O(N^(-2(s-n)/d)L^(-2(s-n)/d)) in the W^n,p norm.
• All constants in the error bounds are explicitly determined, offering precise performance guarantees.

Conclusions:

• The study provides rigorous, nonasymptotic approximation guarantees for DNNs in Sobolev spaces.
• The findings offer theoretical support for using DNNs in numerical PDE solvers.
• Explicit error bounds enable better understanding and control of approximation accuracy based on network architecture.