ReLU Networks Are Universal Approximators via Piecewise Linear or Constant Functions

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Continuous functions are fundamental in mathematics and are often approximated by simpler functions.
• Rectified Linear Unit (ReLU) networks are a core component of modern deep learning architectures.
• Understanding the approximation capabilities of ReLU networks is crucial for advancing deep learning theory.

Purpose of the Study:

• To prove that ReLU networks can approximate any continuous function with arbitrary precision.
• To explore the use of piecewise linear and constant approximations for function approximation using ReLU networks.
• To introduce a new neural unit (TRLU) for implementing piecewise constant approximations.

Main Methods:

• For univariate functions, ReLUs are composed to create piecewise linear approximations.
• For multivariate functions, triangulation methods are used to derive piecewise linear functions approximated by ReLU networks.
• A novel neural unit, TRLU, is designed using ReLU networks to achieve piecewise constant approximations, akin to Haar wavelets.

Main Results:

• The study provides a theoretical proof for the approximation capabilities of ReLU networks.
• Piecewise linear approximation is achieved for univariate and multivariate functions.
• The TRLU unit successfully implements piecewise constant approximations.

Conclusions:

• ReLU networks are powerful tools for approximating continuous functions with high precision.
• The proposed methods offer new ways to construct and interpret deep neural networks.
• The TRLU unit expands the applicability of ReLU networks for tasks requiring piecewise constant representations.