Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) often suffers from Q-value overestimations due to function approximation errors, hindering policy performance.
• Mitigating these overestimations is crucial for improving the effectiveness of RL algorithms in complex control tasks.

Purpose of the Study:

• To present a novel algorithm, distributional soft actor-critic (DSAC), designed to mitigate Q-value overestimations in off-policy RL for continuous control.
• To theoretically and empirically demonstrate the benefits of learning return distributions for enhancing policy performance.

Main Methods:

• Developed a distributional soft policy iteration (DSPI) framework by integrating return distribution learning into maximum entropy RL.
• Introduced DSAC, a deep off-policy actor-critic variant of DSPI, which learns continuous return distributions and manages their variance to prevent gradient issues.

Main Results:

• The proposed DSAC algorithm effectively mitigates Q-value overestimations by adaptively adjusting the Q-value function's update step size.
• DSAC achieved state-of-the-art performance on the suite of MuJoCo continuous control tasks.

Conclusions:

• Learning the distribution of state-action returns is a theoretically sound approach to address Q-value overestimations in RL.
• DSAC offers a practical and effective solution for improving policy performance in continuous control settings, demonstrating superior results on benchmark tasks.