Limitations of shallow nets approximation.

Area of Science:

• Machine Learning
• Functional Analysis
• Deep Learning Theory

Background:

• Reproducing Kernel Hilbert Spaces (RKHS) are fundamental in machine learning for function approximation.
• Shallow neural networks are widely used but their theoretical approximation capabilities are not fully understood compared to deep networks.
• Understanding approximation limits is crucial for designing efficient neural network architectures.

Purpose of the Study:

• To analyze the approximation abilities of shallow neural networks in RKHS.
• To establish a theoretical lower bound for approximation errors achievable by shallow networks.
• To provide a comparative analysis explaining the performance advantage of deep networks over shallow networks.

Main Methods:

• Mathematical analysis of function approximation using shallow neural networks.
• Development of a probability measure to define achievable approximation bounds.
• Comparison of derived bounds with classical minimax approximation error estimates.

Main Results:

• A probability measure was identified, enabling the realization of achievable lower bounds for shallow network approximation.
• The derived bounds differ from classical minimax approximation error estimates.
• The findings highlight inherent limitations of shallow networks in approximating functions within RKHS balls.

Conclusions:

• Shallow networks exhibit specific limitations in approximating functions within RKHS, particularly compared to deep networks.
• The theoretical explanation supports the empirical observation that deep networks often achieve better performance.
• This research contributes to the theoretical understanding of neural network expressivity and architectural advantages.