A3C-GS: Adaptive Moment Gradient Sharing With Locks for Asynchronous Actor-Critic Agents.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Parallel actor-critic algorithms are crucial for complex reinforcement learning tasks.
• Existing methods often require explicit entropy regularization for effective exploration.
• Exploration remains a challenge in high-dimensional and sparse reward environments.

Purpose of the Study:

• To propose an asynchronous gradient sharing mechanism (A3C-GS) for parallel actor-critic algorithms.
• To improve exploration characteristics without relying heavily on entropy loss terms.
• To ensure long-term convergence to optimal policies while enhancing short-term exploration.

Main Methods:

• Developed an asynchronous gradient sharing mechanism as a composition of two contractions: gradient computation and lock-coordinated gradient sharing.
• The mechanism induces temporary policy heterogeneity for exploration and ensures convergence under specific conditions.
• Analyzed the theoretical properties of the gradient sharing operation for short- and long-term behavior.

Main Results:

• A3C-GS automatically diversifies worker policies for improved exploration, reducing the need for entropy loss.
• The algorithm theoretically guarantees convergence to the optimal policy with a small learning rate and gradient clipping.
• Empirical results in high-dimensional, sparse reward, and 3-D navigation tasks show superior performance compared to the base A3C algorithm.

Conclusions:

• The proposed A3C-GS mechanism offers enhanced exploration and stable convergence in parallel actor-critic reinforcement learning.
• It effectively addresses challenges in high-dimensional environments and sparse reward settings.
• A3C-GS represents a significant improvement over standard asynchronous advantage actor-critic (A3C) methods.