Enhancing PPO With Trajectory-Aware Hybrid Policies.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Proximal policy optimization (PPO) is a leading on-policy algorithm in reinforcement learning.
• PPO demonstrates stable performance but suffers from high variance and sample complexity.
• These limitations hinder its application in complex continuous control tasks.

Purpose of the Study:

• To introduce a novel hybrid-policy PPO (HP3O) algorithm.
• To address the high variance and sample complexity issues in PPO.
• To enhance the efficiency and performance of reinforcement learning agents.

Main Methods:

• HP3O employs a trajectory replay buffer utilizing a first-in, first-out (FIFO) strategy.
• Recent trajectories are stored to mitigate data distribution drift.
• Policy updates use a batch comprising the best-performing trajectory and randomly sampled ones.

Main Results:

• HP3O demonstrated empirical variance reduction.
• The algorithm showed improved performance over baseline methods in continuous control environments.
• Theoretical policy improvement guarantees were established for HP3O.

Conclusions:

• HP3O effectively utilizes recent trajectories to improve policy learning.
• The proposed method offers a promising approach to enhance PPO's efficiency and stability.
• HP3O represents a significant advancement for reinforcement learning in continuous control domains.