Absorbing state dynamics of stochastic gradient descent

Area of Science:

• Statistical Physics
• Machine Learning
• Optimization

Background:

• Stochastic gradient descent (SGD) is a core algorithm in machine learning and optimization.
• Understanding SGD's behavior, especially in complex energy landscapes, is crucial for improving model performance.
• Existing models often lack a direct physical interpretation of SGD's dynamics.

Purpose of the Study:

• To investigate Stochastic Gradient Descent (SGD) through the framework of particle dynamics.
• To establish a link between SGD and models from nonequilibrium statistical physics.
• To analyze the phase transition behavior of SGD and its relation to physical systems.

Main Methods:

• Developed a minimal physical model of SGD using spherical particles that minimize energy by reducing overlaps.
• Employed the biased random organization (BRO) model, a nonequilibrium absorbing state model, to describe SGD dynamics.
• Analyzed particle interactions, noise mechanisms, and critical phenomena near phase transitions.

Main Results:

• SGD dynamics were shown to be equivalent to BRO dynamics for particles with linear repulsive interactions under specific noise conditions.
• Both SGD and BRO models exhibited behavior consistent with the Manna universality class near a critical packing fraction (ϕ_{c}≈0.64).
• SGD was observed to favor flatter energy landscape regions, correlating with solutions that generalize better in neural network training.

Conclusions:

• A novel physical framework connects SGD to nonequilibrium statistical physics via particle dynamics.
• The study provides insights into SGD's optimization process and its tendency to find generalizable solutions.
• The findings suggest potential for new optimization strategies inspired by physical models.