Conformal Symplectic Optimization for Stable Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization Algorithms

Background:

• Deep reinforcement learning (RL) training is hindered by unstable, non-convex stochastic optimization.
• Existing methods struggle with the inherent trial-and-error instability in RL agent training.

Purpose of the Study:

• Introduce relativistic adaptive gradient descent (RAD), a novel physics-inspired optimization algorithm.
• Enhance long-term training stability and performance in deep reinforcement learning.

Main Methods:

• Conceptualize neural network (NN) training as a conformal Hamiltonian system evolution.
• Utilize relativistic kinetic energy to limit parameter update speeds, inspired by special relativity.
• Model NN optimization as a multiparticle system with adaptive learning rates for each parameter.

Main Results:

• Prove RAD's sublinear convergence in general non-convex settings.
• Demonstrate RAD outperforms nine baseline optimizers across five RL algorithms and twelve environments.
• Achieve up to 155.1% performance improvement over ADAM in Atari games.

Conclusions:

• RAD offers a universal framework for transferring long-term stability to iterative NN updating rules.
• The algorithm effectively mitigates abnormal gradient influences, stabilizing and accelerating RL training.
• RAD provides a significant advancement over existing optimizers like ADAM for deep reinforcement learning.