Absorbing state dynamics of stochastic gradient descent

Area of Science:

• Statistical Physics
• Machine Learning
• Stochastic Optimization

Background:

• Stochastic gradient descent (SGD) is a cornerstone of machine learning and stochastic optimization.
• Understanding SGD's behavior is crucial for advancing AI and complex system modeling.
• Existing frameworks often lack direct links to physical processes.

Purpose of the Study:

• To investigate Stochastic Gradient Descent (SGD) through the framework of particle dynamics.
• To establish a connection between SGD and models from nonequilibrium statistical physics.
• To analyze the phase transition behavior of SGD in a simplified physical model.

Main Methods:

• Developed a minimal physical model of spherical particles undergoing SGD to minimize energy.
• Employed the biased random organization (BRO) model, a nonequilibrium absorbing state model, to describe SGD dynamics.
• Analyzed particle interactions, noise mechanisms, and critical phenomena near phase transitions.

Main Results:

• SGD dynamics were shown to be approximable by particles with linear repulsive interactions under anisotropic noise.
• In the small learning rate limit, SGD and BRO dynamics for repulsive particles become equivalent, converging to a critical packing fraction (ϕ_{c}≈0.64).
• Both models exhibit critical behavior consistent with the Manna universality class.

Conclusions:

• SGD's behavior can be effectively described using concepts from nonequilibrium statistical physics, specifically BRO dynamics.
• The equivalence between SGD and BRO near criticality provides a novel physical interpretation of optimization processes.
• SGD's tendency to favor flatter energy landscapes, observed above the transition, correlates with improved generalization in neural network training.