Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation

Area of Science:

• Computational Neuroscience
• Machine Learning
• Statistical Physics

Background:

• Deep feed-forward neural networks (NN) are powerful tools for information processing.
• Understanding the dynamics of learning in deep NNs, especially in the continuum depth limit, remains a challenge.
• The Renormalization Group (RG) is a theoretical framework used in physics to describe systems at different scales.

Purpose of the Study:

• To investigate the information processing dynamics in deep feed-forward neural networks in the continuum depth limit.
• To establish an analogy between neural network dynamics and the Renormalization Group (RG) framework.
• To provide a new perspective on Bayesian learning and its connection to statistical physics.

Main Methods:

• Encoding neural network weights into a Maximum Entropy (Maxent) family of distributions.
• Utilizing Bayesian learning to update probability distributions and analyze hyper-parameter changes.
• Deriving a diffusion-like partial differential equation (PDE) analogous to Wilson's RG in the continuum limit.
• Recasting the learning dynamics in the language of dynamical programming and Hamilton-Jacobi-Bellman equations.

Main Results:

• The information processing dynamics of deep neural networks in the continuum depth limit can be described using Renormalization Group (RG) language.
• Neural network concept association is analogous to RG's identification of key variables characterizing thermodynamic states.
• The learning dynamics exhibit an entropic dynamic where hyper-parameters follow the gradient of the evidence, leading to a PDE similar to RG.

Conclusions:

• A strong analogy exists between the learning dynamics of deep neural networks and the Renormalization Group (RG).
• This connection provides a novel framework for understanding information processing in artificial neural networks through the lens of statistical physics.
• The study offers a new perspective on Bayesian learning and its mathematical description, potentially impacting future AI and physics research.