Actor-Critic With Synthesis Loss for Solving Approximation Biases.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Approximation biases in value functions, specifically overestimation and underestimation, are significant limitations in current reinforcement learning (RL) algorithms.
• These biases arise from value mismatching between actual returns and action-value approximations, hindering RL performance.

Purpose of the Study:

• To develop a novel synthesis loss function to mitigate approximation biases in RL's action-value estimation.
• To introduce a new discrepancy function for identifying and quantifying approximation biases.
• To propose a new actor-critic (AC) algorithm, ACSL, integrating the synthesis loss and an error-controlled mechanism.

Main Methods:

• A new synthesis loss function is developed, incorporating a regularization term and a modified clipped double Q-learning structure.
• A novel discrepancy function is introduced to precisely determine the type and magnitude of approximation biases.
• Two coefficients within the synthesis loss are automatically tuned by minimizing the discrepancy function during training.
• A new actor-critic algorithm, ACSL, is designed by integrating the synthesis loss and an error-controlled mechanism.

Main Results:

• The proposed ACSL algorithm demonstrates superior performance compared to state-of-the-art RL methods on various continuous control tasks.
• The synthesis loss function effectively reduces approximation biases and enhances overall performance.
• The synthesis loss function is easily adaptable to other RL algorithms, improving their efficacy.

Conclusions:

• The developed synthesis loss function and ACSL algorithm effectively address approximation biases in RL.
• The proposed methods offer significant improvements in performance for complex continuous control tasks.
• The synthesis loss function presents a valuable and easily implementable tool for enhancing existing RL algorithms.