Boosting On-Policy Actor-Critic With Shallow Updates in Critic

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep reinforcement learning (DRL) utilizes deep neural networks (NNs) for function approximation.
• Batch reinforcement learning (BRL) offers stable training and data efficiency with fixed representations.

Purpose of the Study:

• To propose Least-Squares Deep Policy Gradient (LSDPG), a hybrid method combining BRL and DRL.
• To achieve enhanced stability and data efficiency by integrating least-squares methods with online DRL.

Main Methods:

• LSDPG employs a shared network for feature sharing between policy (actor) and value function (critic).
• It utilizes regularized least-squares temporal difference (LSTD) for policy evaluation in a stationary critic setting.
• An auxiliary task distills critic features into the representation for improved learning.

Main Results:

• The critic converges to a regularized TD fixpoint, and the actor converges to a locally optimal policy under specific conditions.
• LSDPG demonstrates improved sample efficiency compared to Proximal Policy Optimization and Phasic Policy Gradient on the Procgen benchmark.

Conclusions:

• LSDPG effectively combines the strengths of BRL and DRL, offering a stable and data-efficient approach.
• The method shows promise for improving performance in complex reinforcement learning environments.