Dimension independent bounds for general shallow networks

Area of Science:

• Machine Learning
• Function Approximation
• Manifold Learning

Background:

• Function approximation faces challenges with high-dimensional data (curse of dimensionality).
• Estimating out-of-sample approximation for manifold learning is crucial.
• Shallow networks (neural networks, RBF networks, kernels) are used for function approximation on manifolds.

Purpose of the Study:

• To develop a unified abstract theorem for function approximation.
• To address the curse of dimensionality and estimate approximation degrees in manifold learning.
• To analyze the approximation capabilities of shallow and deep networks.

Main Methods:

• Proving dimension-independent approximation bounds for G-networks on compact metric measure spaces.
• Defining a generalized notion of dimension based on maximal distinguishable sets.
• Analyzing deep networks as compositions of shallow networks via directed acyclic graphs.

Main Results:

• Dimension-independent bounds for shallow networks (G-networks) were established.
• Bounds improve with kernel smoothness, offering better approximation without saturation.
• Estimates for out-of-sample extension in manifold learning were derived.

Conclusions:

• Deep networks with non-smooth activations (e.g., ReLU) do not inherently outperform shallow networks in approximation degree.
• The abstract theorem provides a unified framework for function approximation and manifold learning challenges.
• The findings highlight the importance of network architecture and activation functions in approximation performance.