Smooth Function Approximation by Deep Neural Networks with General Activation Functions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning Theory

Background:

• Growing interest in the expressivity of deep neural networks (DNNs).
• Existing research primarily focuses on specific activation functions like ReLU and sigmoid.
• Lack of comprehensive analysis for a broader class of activation functions.

Purpose of the Study:

• To investigate the approximation capabilities of DNNs with a wide range of activation functions.
• To determine the necessary network depth, width, and sparsity for approximating Hölder smooth functions.
• To establish the theoretical performance limits (minimax optimality) of DNNs with general activation functions.

Main Methods:

• Derivation of approximation error bounds for DNNs.
• Analysis of network depth, width, and sparsity requirements.
• Theoretical framework for minimax optimality in regression and classification.

Main Results:

• Quantified the depth, width, and sparsity needed for DNNs to approximate Hölder smooth functions within a given error.
• Established theoretical guarantees for the approximation ability of DNNs across a broad class of activation functions.
• Demonstrated the minimax optimality of DNN estimators for general activation functions.

Conclusions:

• The findings provide a generalized understanding of DNN expressivity.
• The derived theoretical results are applicable to various activation functions, enhancing DNN design and analysis.
• This work contributes to the foundational theory of deep learning, impacting both regression and classification tasks.