Scale-invariant feature extraction of neural network and renormalization group flow

Area of Science:

• Statistical Physics
• Machine Learning
• Deep Learning

Background:

• The feature extraction process in deep neural networks (DNNs) is not fully understood but is thought to be hierarchical, resembling coarse-graining.
• The renormalization group (RG) concept from statistical physics describes coarse-graining and scale transformations.

Purpose of the Study:

• To investigate potential relationships between DNNs and RG by applying Restricted Boltzmann Machines (RBMs) to an Ising model.
• To analyze the parameter flow generated by RBMs and its implications for feature extraction.

Main Methods:

• Utilized an unsupervised Restricted Boltzmann Machine (RBM) trained on spin configurations of an Ising model at various temperatures (T=0 to T=6).
• Constructed a flow of model parameters, specifically temperature, generated by the RBM.
• Analyzed the properties of the weight matrices of the trained RBM.

Main Results:

• The RBM generated a parameter flow where the temperature converged towards the critical value (Tc=2.27) of the Ising model.
• This observed flow is contrary to the typical RG flow direction in the Ising model.
• Analysis of RBM weight matrices provided insights into the mechanism driving the flow towards Tc and feature learning.

Conclusions:

• Unsupervised RBMs trained on Ising models exhibit a parameter flow that approaches the critical temperature, offering a novel perspective on the connection between machine learning and statistical physics.
• The RBM's ability to learn features is linked to its tendency to flow towards the critical temperature, suggesting a mechanism for hierarchical feature extraction.
• This research opens avenues for exploring RG concepts in understanding deep learning architectures and their feature learning capabilities.