Actor-Critic Learning Control Based on -Regularized Temporal-Difference Prediction With Gradient Correction.

Area of Science:

• Reinforcement Learning
• Machine Learning
• Control Theory

Background:

• Actor-critic (AC) methods based on policy gradient (PG-based AC) are prevalent for learning control problems.
• Enhancing data efficiency in the critic component of PG-based AC has led to research in recursive least-squares temporal difference (RLS-TD) algorithms for policy evaluation.
• Existing RLS-TD critic implementations evaluate mixed policies from varying actors, hindering convergence proofs to optimal fixed points.

Purpose of the Study:

• To propose a new actor-critic framework, Critic-Iteration Policy Gradient (CIPG), that addresses the convergence limitations of existing RLS-TD critic methods.
• To enable on-policy learning of the state-value function for the current policy.
• To achieve gradient ascent towards maximizing the discounted total reward.

Main Methods:

• CIPG maintains fixed policy parameters within each iteration.
• It employs an RLS-TD critic with -regularization for evaluating the fixed policy.
• Convergence analysis is extended for PG with function approximation to incorporate the RLS-TD critic.

Main Results:

• The -regularization term in the CIPG critic remains active throughout the learning process.
• CIPG demonstrates superior learning efficiency compared to conventional AC methods.
• Simulation results indicate a faster convergence rate for CIPG.

Conclusions:

• CIPG provides a theoretically sound and practically effective framework for policy gradient actor-critic methods.
• The proposed method overcomes convergence issues associated with RLS-TD critics evaluating non-fixed policies.
• CIPG offers improved performance in learning control tasks.