Reinforcement Learning From Hierarchical Critics.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) agents often struggle with complex competition tasks due to limited information.
• Existing actor-critic methods may not fully leverage global context for improved decision-making.
• Hierarchical structures can potentially enhance learning efficiency and performance in multi-agent systems.

Purpose of the Study:

• To develop and evaluate a novel reinforcement learning algorithm, RL from hierarchical critics (RLHC), for competitive tasks.
• To investigate the impact of hierarchical global information on learning speed and cumulative rewards.
• To compare the performance of RLHC against a benchmark algorithm in diverse competition scenarios.

Main Methods:

• An actor-critic RL framework was extended with multiple cooperative critics organized in a two-level hierarchy.
• The proposed RLHC algorithm enables agents to access both local and global value information through the hierarchical critics.
• Agents utilize top-down coordination from higher-level critics to integrate global information into their learning process.

Main Results:

• The RLHC algorithm demonstrated superior performance compared to the proximal policy optimization (PPO) benchmark.
• Significant improvements in training performance and cumulative rewards were observed across four distinct competition tasks.
• The effectiveness of integrating hierarchical global information was validated in simulated environments.

Conclusions:

• The RLHC algorithm effectively utilizes hierarchical global information to enhance reinforcement learning in competitive settings.
• This approach offers a promising direction for developing more capable and efficient intelligent agents in complex multi-agent scenarios.
• RLHC provides a robust method for improving learning speed and overall performance in competitive AI tasks.