Cooperative modular reinforcement learning for large discrete action space problem

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Deep Reinforcement Learning (DRL) excels in high-dimensional tasks but struggles with convergence and sample efficiency in large discrete action spaces.
• Existing DRL methods face significant challenges in effectively handling problems with a vast number of possible actions.

Purpose of the Study:

• To develop a novel Cooperative Modular Reinforcement Learning (CMRL) method for efficiently solving problems with large discrete action spaces.
• To enhance training efficiency and maintain solution accuracy compared to existing Deep Reinforcement Learning approaches.

Main Methods:

• Introduced a task decomposition method using rule-based action division to break down complex decisions into smaller sub-tasks.
• Proposed a CMRL framework with multiple Critic networks, each learning a decomposed value function for local optimal actions within sub-tasks.
• Implemented a parallel training mechanism for training multiple Critic networks concurrently with diverse models and data.

Main Results:

• CMRL demonstrated superior training efficiency compared to classical and state-of-the-art Deep Reinforcement Learning methods.
• The method successfully maintained solution accuracy while significantly improving performance on large action space problems.
• Experimental validation across four simulations confirmed the generality and effectiveness of CMRL.

Conclusions:

• CMRL offers a robust and efficient solution for tackling large discrete action space problems in reinforcement learning.
• The cooperative modular approach and parallel training mechanism provide significant advantages over traditional DRL methods.