A Non-equilibrium thermodynamic framework for neural networks: A principled correspondence and parameter dynamics

Deep learning's opaque training dynamics preclude a unified explanation for parameter evolution and the nature of generalization. Existing physics-inspired approaches often exacerbate this issue by overlooking core thermodynamic concerns such as energy conversion and entropy costs. To forge this missing quantitative link, this article posits that the training process is dynamically correspondent under the Langevin approximation and quasi-equilibrium interpretation to a non-equilibrium thermodynamic system. We therefore construct a unified theoretical framework, which we term the Neural Networks' Thermodynamic Framework (NNTF), conceptualizing network parameters as effective particles and grounding the system in Boltzmann thermodynamics by defining three key state functions: Internal Energy, representing generalization potential; Entropy, measuring parameter disorder; and Temperature, quantifying parameter exploration. This framework yields three fundamental principles for deep learning: 1) the Principle I: Conservation of Neural Internal Energy, quantifying the data-computation trade-off to inform strategies for minimum computational work; 2) the Principle II: Entropy Production in Neural Training, linking entropy production and free-energy curvature to training-only instability diagnostics; and 3) the Principle III: Inaccessibility of Ideal Generalization, connecting performance-floor reasoning with a training-only diminishing-returns stopping criterion. Experiments on CIFAR-10/100 show training-only instability and stopping diagnostics with variable timing separations and budget-saving diminishing returns, while AdamW-trained Vision Transformer stress tests suggest sensitivity to sharp-minima-ejection signals in anisotropic, attention-based regimes. This work provides a thermodynamically motivated perspective on deep learning theory, enhancing interpretability and enabling the design of energy-efficient training protocols.