How neural networks work: Unraveling the mystery of randomized neural networks for functions and chaotic dynamical...

Artificial Neural Networks (ANNs) have proven to be fantastic at a wide range of machine learning tasks, and they have certainly come into their own in all sorts of technologies that are widely consumed today in society as a whole. A basic task of machine learning that neural networks are well suited to is supervised learning, including when learning orbits from time samples of dynamical systems. The usual construct in ANN is to fully train all of the perhaps many millions of parameters that define the network architecture. However, there are certain ANN algorithms that work well with random designs. We have previously presented an explanation as to how the reservoir computing recurrent neural network architecture succeeds despite randomness. Here, we explain how the random feedforward neural networks called the random project networks work. In particular, we present examples for both general function learning and also for learning a flow from samples of orbits of chaotic dynamical systems. There is an interesting geometric explanation of the success, in the case of the ReLu activation function, that relates to the classical mathematical question of how configurations of random lines fall in a plane, or how planes or hyperplanes may fall in higher dimensional spaces. These random configurations lead to a refinement of the domain so that piecewise linear continuous functions result that are dense in continuous functions. This relates neural networks to finite element methods. We highlight the universality of this representation by forecasting the skill of chaotic dynamical systems.