A Non-equilibrium thermodynamic framework for neural networks: A principled correspondence and parameter dynamics

Area of Science:

• Thermodynamics
• Machine Learning
• Deep Learning Theory

Background:

• Deep learning training dynamics are opaque, hindering understanding of parameter evolution and generalization.
• Existing physics-inspired methods neglect crucial thermodynamic principles like energy conversion and entropy costs.

Purpose of the Study:

• To establish a quantitative link between deep learning training and non-equilibrium thermodynamics.
• To introduce the Neural Networks' Thermodynamic Framework (NNTF) for a unified theoretical perspective.

Main Methods:

• Conceptualizing network parameters as effective particles within Boltzmann thermodynamics.
• Defining key state functions: Internal Energy (generalization potential), Entropy (parameter disorder), and Temperature (parameter exploration).
• Formulating three core principles: Conservation of Neural Internal Energy, Entropy Production in Neural Training, and Inaccessibility of Ideal Generalization.

Main Results:

• Demonstrated training-only instability and stopping diagnostics on CIFAR-10/100 datasets.
• Identified budget-saving diminishing returns and sensitivity to sharp-minima-ejection signals in Vision Transformers.
• Validated thermodynamically motivated diagnostics for training efficiency and stability.

Conclusions:

• The NNTF provides a thermodynamically grounded perspective on deep learning.
• Enhances interpretability of deep learning models and training processes.
• Enables the design of more energy-efficient and stable deep learning training protocols.