Neural networks trained by weight permutation are universal approximators.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• The universal approximation property is key to neural network success, typically achieved through unconstrained parameter training.
• Recent experimental work introduced permutation-based training, achieving classification performance without modifying exact weight values.

Purpose of the Study:

• To provide a theoretical guarantee for the permutation training method.
• To demonstrate its capability in guiding ReLU networks for function approximation.

Main Methods:

• Theoretical analysis to prove the approximation capability of permutation training for one-dimensional continuous functions.
• Numerical experiments on regression tasks with diverse initializations to validate the method's efficiency.

Main Results:

• Theoretical proof confirming that permutation training can guide ReLU networks to approximate one-dimensional continuous functions.
• Empirical validation of the method's efficiency in regression tasks across various initializations.
• Observations suggesting permutation training as a tool for understanding network learning behavior.

Conclusions:

• Permutation training provides a theoretical basis for approximating continuous functions with ReLU networks.
• The method is efficient for regression tasks and offers novel insights into neural network learning dynamics.