Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective

Area of Science:

• Machine Learning
• Deep Learning Theory
• High-Dimensional Geometry

Background:

• Deep neural networks (DNNs) are known to approximate any target function given sufficient width.
• Existing theoretical explanations are often analytically complex.

Purpose of the Study:

• To provide an elementary geometrical proof for the function approximation capabilities of DNNs.
• To elucidate the underlying structure using a simplified model.

Main Methods:

• Utilized a simple model of a randomly connected deep network.
• Employed geometrical analysis in high-dimensional spaces.
• Investigated the projection of a high-dimensional sphere onto a lower-dimensional subspace.

Main Results:

• Demonstrated that high-dimensional geometry plays a crucial role in DNN function realization.
• Showed that projecting a uniform distribution from a high-dimensional sphere to a low-dimensional subspace results in a Gaussian distribution with minimal variance and covariance.

Conclusions:

• The study offers a geometrically intuitive explanation for the expressivity of wide deep neural networks.
• Highlights the significance of geometric transformations in understanding neural network behavior.