Nonlinear approximation via compositions.

Area of Science:

• Applied Mathematics
• Machine Learning
• Numerical Analysis

Background:

• Nonlinear approximation aims to represent functions using a limited number of terms.
• Deep learning has shown success with compositional function representations.
• Feed-forward neural networks (FNNs) are a key tool for implementing these functions.

Purpose of the Study:

• To investigate the benefits of using compositional functions within dictionaries for nonlinear approximation.
• To analyze how the number of hidden layers (L) in FNNs affects approximation rates.
• To compare the efficiency of wide vs. deep FNNs in this context.

Main Methods:

• Proposing dictionaries composed of functions implemented as ReLU FNNs with L hidden layers.
• Analyzing the best N-term approximation rates for different values of L.
• Deriving theoretical bounds for approximation error.
• Considering Hölder continuous functions on [0,1]^d.

Main Results:

• Increasing L from 1 to 2 can double the best N-term approximation rate (from O(N^{-η}) to O(N^{-2η})).
• For L=3, an essentially tight rate of O(N^{-2α/d}) is achieved for Hölder continuous functions.
• Beyond L=3, increasing L does not further improve the approximation rate in terms of N.
• Wide FNNs with few layers are computationally more efficient than deep, narrow FNNs for parallel computing.

Conclusions:

• Compositional dictionaries implemented with FNNs significantly enhance nonlinear approximation capabilities.
• The number of hidden layers in FNNs is crucial, with optimal performance around L=2 or L=3.
• Architectural choices in FNNs impact computational efficiency, favoring wider networks for parallel processing.