A Generic Competitive-Cooperative Actor-Critic Framework for Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep Reinforcement Learning (DRL) faces challenges in exploration and Q-value estimation.
• Double-actor DRL methods show promise but lack actor collaboration, leading to suboptimal policies.

Purpose of the Study:

• To propose a generic solution for promoting mutual learning and collaboration among actors in double-actor DRL methods.
• To improve policy development and overall performance in DRL by addressing actor independence.

Main Methods:

• Introduced a method to minimize the difference in actions output by actors, fostering mutual imitation.
• Incorporated minimizing Q-value differences from critics to ensure consistent value estimation for imitated actions.
• Developed two specific implementations and extended the approach to other DRL methods.

Main Results:

• The proposed method significantly enhances twenty state-of-the-art (SOTA) DRL methods, including double-actor approaches.
• Improvements were observed across eleven diverse tasks, measured by return and other key metrics.
• Demonstrated broader applicability by extending the method beyond double-actor DRL.

Conclusions:

• The proposed mutual learning framework effectively addresses the limitations of independent actors in double-actor DRL.
• This approach offers a significant advancement in DRL by improving policy optimization and performance.
• The method's generic nature and successful extension to other DRL paradigms suggest wide-ranging potential impact.