Gradient dynamics in reinforcement learning.

Area of Science:

• Statistical mechanics
• Reinforcement learning
• Physics-informed machine learning

Background:

• Supervised learning analysis using statistical mechanics is established.
• Reinforcement learning (RL) remains largely unexplored by physicists.

Purpose of the Study:

• Analyze the dynamics of the policy gradient algorithm in reinforcement learning.
• Bridge the gap between statistical mechanics and reinforcement learning.

Main Methods:

• Conducted analysis for a convex k-armed bandit problem.
• Developed a mapping for a non-convex robotic arm problem to a p-spin glass model.
• Utilized Langevin equation to describe learning dynamics.

Main Results:

• Policy gradient learning dynamics follow drift-diffusion motion (Langevin equation).
• Learning rate tunes coefficients of the Langevin equation.
• Learning rate acts as an effective temperature in non-convex settings, smoothing landscapes.

Conclusions:

• Demonstrated a physics-inspired approach to understanding and optimizing RL algorithms.
• Highlighted similarities between RL dynamics and genotype evolution (Kimura equation).
• Proposed annealing procedures for disordered systems to optimize RL.