Efficient Approximation of High-Dimensional Functions With Neural Networks

Area of Science:

• Machine Learning
• Numerical Analysis
• Computer Science

Background:

• The curse of dimensionality poses a significant challenge in high-dimensional approximation problems, limiting the efficiency of traditional methods.
• Neural networks have shown promise in overcoming these limitations, but their application in high dimensions requires further investigation.
• Understanding the approximation capabilities of neural networks for complex functions is crucial for advancing computational mathematics.

Purpose of the Study:

• To develop a theoretical framework demonstrating how neural networks can surmount the curse of dimensionality in approximation tasks.
• To introduce and analyze 'catalog networks,' a generalized neural network architecture with adaptable activation functions.
• To establish conditions under which catalog networks can be efficiently approximated by rectified linear unit (ReLU) networks.

Main Methods:

• The study proposes a framework based on 'catalog networks,' where activation functions can vary across layers from a predefined set.
• Theoretical analysis is employed to establish the approximation capabilities of catalog networks.
• The research investigates the efficient approximation of catalog networks using rectified linear unit (ReLU)-type networks, providing parameter estimates.

Main Results:

• Catalog networks are shown to be a rich family of continuous functions capable of overcoming the curse of dimensionality.
• Under specific conditions on the function catalog, catalog networks can be efficiently approximated by ReLU networks.
• Precise estimates for the number of parameters required for a given approximation accuracy are provided.

Conclusions:

• The developed framework confirms that neural networks, specifically catalog networks, can effectively address the curse of dimensionality.
• Rectified linear unit networks can efficiently approximate catalog networks, leading to classes of functions that avoid the curse of dimensionality.
• This research offers a theoretical foundation for designing neural network architectures that are scalable and efficient in high-dimensional spaces.