Gradient Monitored Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep reinforcement learning (RL) algorithms often face challenges with slow convergence and limited generalization.
• High variance in gradients during training can hinder efficient learning and lead to suboptimal policy.
• Existing methods may not dynamically adapt to the learning process for optimal performance.

Purpose of the Study:

• To present a novel neural network training approach, gradient monitoring (GM), for enhancing deep reinforcement learning performance.
• To reduce gradient variance for a more targeted and efficient learning process.
• To introduce adaptive variants of GM that automatically adjust learning strategies and network architecture.

Main Methods:

• Gradient Monitoring (GM): A method to steer neural network weight updates based on dynamic training feedback.
• Momentum with GM (M-WGM): Adjusts backpropagated gradients based on learning parameters.
• Adaptive M-WGM (AM-WGM): Dynamically adjusts learning focus and automatically determines network size by freezing weights.

Main Results:

• Proposed GM variants demonstrably improve model performance and generalization capabilities.
• AM-WGM facilitates automatic adjustment between focused and dispersed learning based on reward feedback.
• The method successfully applied to discrete (multi-robot coordination, Atari games) and continuous (MuJoCo) control tasks using A2C and PPO.

Conclusions:

• Gradient monitoring offers a significant advancement in training deep reinforcement learning models.
• The adaptive nature of AM-WGM enhances learning efficiency and automates network architecture selection.
• The presented methods show strong applicability and performance improvements, particularly in generalization for RL agents.