Anti-Martingale Proximal Policy Optimization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• On-policy deep reinforcement learning (DRL) requires high sample efficiency due to single-use data for parameter updates.
• Accelerating DRL training necessitates improved methods for utilizing exploration data.

Purpose of the Study:

• To develop a novel framework for efficient sample selection in on-policy DRL.
• To enhance the training speed and performance of DRL algorithms.

Main Methods:

• Proposed a submartingale criterion based on the optimal policy-martingale equivalence.
• Developed an advanced value iteration (AVI) method for accurate value iteration.
• Introduced an anti-martingale (AM) reinforcement learning framework for effective sample selection.
• Integrated the AM framework with proximal policy optimization (PPO) into the AM proximal policy optimization (AMPPO) method.

Main Results:

• AMPPO accelerates the state value updating process while adhering to the submartingale criterion.
• Experimental results on the Mujoco platform demonstrate superior performance of AMPPO compared to state-of-the-art DRL methods.

Conclusions:

• The proposed AMPPO method significantly improves sample efficiency in on-policy DRL.
• AMPPO offers a promising approach for accelerating DRL training and achieving better performance.