Double actor-critic with TD error-driven regularization in reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Value estimation is crucial for reinforcement learning (RL) performance.
• Existing actor-critic methods can suffer from estimation inaccuracies.
• Improving RL efficiency and stability remains an active research area.

Purpose of the Study:

• To propose a novel algorithm, TDDR, for enhanced value estimation in reinforcement learning.
• To leverage the benefits of double actor-critic frameworks with novel regularization.
• To simplify the implementation of advanced RL algorithms.

Main Methods:

• Developed TDDR, a double actor-critic algorithm with temporal difference error-driven regularization.
• Utilized double actors, each paired with a critic, to maximize the advantages of double critics.
• Introduced an innovative critic regularization architecture.

Main Results:

• TDDR demonstrated superior value estimation compared to classical deterministic policy gradient methods.
• The algorithm achieved competitive performance against 13 other algorithms on MuJoCo and Box2D tasks.
• TDDR showed statistically significant performance improvements in several experimental environments.

Conclusions:

• TDDR offers a simplified yet effective approach to reinforcement learning value estimation.
• The algorithm's convergence properties were analyzed under different updating patterns.
• TDDR represents a significant advancement in actor-critic frameworks for RL.