Related Experiment Video
Updated: Nov 27, 2025
11:03
An Analog Macroscopic Technique for Studying Molecular Hydrodynamic Processes in Dense Gases and Liquids
Published on: December 4, 2017
8.8K
Entropy, Free Energy, and Work of Restricted Boltzmann Machines
Sangchul Oh1, Abdelkader Baggag2, Hyunchul Nha3
1Qatar Environment and Energy Research Institute, Hamad Bin Khalifa University, Qatar Foundation, 5825 Doha, Qatar.
Entropy (Basel, Switzerland)
|December 8, 2020
Summary
This study analyzes restricted Boltzmann machine training using statistical physics. We show growing correlations during training and use Monte-Carlo simulations to calculate work, connecting it to free energy differences.
Area of Science:
- Statistical Physics
- Machine Learning
- Computational Neuroscience
Background:
- Restricted Boltzmann Machines (RBMs) are generative probabilistic models.
- RBMs utilize the Boltzmann distribution to define probabilities of network configurations.
- Network learning involves optimizing energy function parameters based on training data.
Purpose of the Study:
- To analyze the RBM training process through the lens of statistical physics.
- To illustrate thermodynamic quantities during RBM training.
- To investigate the relationship between work, free energy, and training dynamics.
Main Methods:
- Calculation of thermodynamic quantities (entropy, free energy, internal energy) as a function of training epochs.
- Monte-Carlo simulations of visible and hidden vector trajectories in configuration space.
- Analysis of the subadditivity of entropies to demonstrate layer correlations.
Main Results:
- Demonstrated growth of correlation between visible and hidden layers during training via entropy subadditivity.
- Calculated the distribution of work done on the RBM by parameter switching.
- Illustrated thermodynamic quantities like entropy and free energy evolution during training.
Conclusions:
- RBM training exhibits characteristics alignable with statistical physics principles.
- Jarzynski equality provides a link between calculated work and free energy differences.
- The study offers insights into the physical interpretation of machine learning model training.
Related Concept Videos
01:18Entropy
3.2K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.2K
02:39Entropy
33.6K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
33.6K
02:38Third Law of Thermodynamics
20.9K
A pure, perfectly crystalline solid possessing no kinetic energy (that is, at a temperature of absolute zero, 0 K) may be described by a single microstate, as its purity, perfect crystallinity,and complete lack of motion means there is but one possible location for each identical atom or molecule comprising the crystal (W = 1). According to the Boltzmann equation, the entropy of this system is zero.
20.9K
02:39Gibbs Free Energy
36.7K
One of the challenges of using the second law of thermodynamics to determine if a process is spontaneous is that it requires measurements of the entropy change for the system and the entropy change for the surroundings. An alternative approach involving a new thermodynamic property defined in terms of system properties only was introduced in the late nineteenth century by American mathematician Josiah Willard Gibbs. This new property is called the Gibbs free energy (G) (or simply the free...
36.7K
01:05An Introduction to Free Energy
10.1K
How can we compare the energy that releases from one reaction to that of another reaction? We use a measurement of free energy to quantitate these energy transfers. Scientists call this free energy Gibbs free energy (abbreviated with the letter G) after Josiah Willard Gibbs, the scientist who developed the measurement. According to the second law of thermodynamics, all energy transfers involve losing some energy in an unusable form such as heat, resulting in entropy. Gibbs free energy...
10.1K
01:20Entropy and the Second Law of Thermodynamics
3.9K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
3.9K