Dual Parallel Policy Iteration With Coupled Policy Improvement.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Policy iteration (PI) algorithms are fundamental in reinforcement learning.
• Improving the efficiency and convergence of PI remains an active research area.
• Existing PI methods can suffer from slow convergence and suboptimal policy discovery.

Purpose of the Study:

• To develop a novel coupled policy improvement mechanism for PI algorithms.
• To introduce Dual Parallel Policy Iteration (DPPI) for enhanced performance.
• To propose a new reinforcement learning algorithm, parallel TD3 (PTD3), by integrating DPPI with TD3.

Main Methods:

• Developed a coupled policy improvement mechanism using two parallel PIs.
• Introduced a dominant policy concept to guide policy updates via Kullback-Liebler (KL) divergence constraints.
• Combined DPPI with the twin delayed deep deterministic (TD3) policy gradient algorithm to create PTD3.
• Validated PTD3 on continuous-action control tasks using MuJoCo and OpenAI Gym.

Main Results:

• Guaranteed convergence of DPPI under the coupled policy improvement mechanism.
• Demonstrated that Q-functions of new policies are larger than previous dominant policies under certain conditions.
• PTD3 algorithm showed superior performance compared to state-of-the-art RL algorithms in experiments.
• Accelerated PI process and improved policy learning efficiency.

Conclusions:

• The novel coupled policy improvement mechanism effectively enhances PI algorithms.
• DPPI offers guaranteed convergence and improved learning efficiency.
• PTD3 represents a significant advancement in reinforcement learning for continuous control tasks.