Distributional Soft Actor-Critic With Three Refinements.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-free reinforcement learning (RL) algorithms often suffer from performance degradation due to inaccurate Q-value estimation.
• Overestimation of Q-values can lead to suboptimal policies in complex decision-making tasks.
• Previous work introduced Distributional Soft Actor-Critic (DSACv1) to improve value estimation using Gaussian value distributions, but faced training instability and reward scaling sensitivity.

Purpose of the Study:

• To address the limitations of DSACv1, this paper introduces an enhanced algorithm, DSAC with Three refinements (DSAC-T or DSACv2).
• The primary goal is to improve Q-value estimation accuracy, enhance training stability, and ensure robust performance across various reward scales.

Main Methods:

• DSAC-T incorporates three key refinements: expected value substitution, twin value distribution learning, and variance-based critic gradient adjustment.
• The enhanced algorithm was systematically evaluated on a diverse set of benchmark tasks and a real-world robotic control application.

Main Results:

• DSAC-T consistently matches or outperforms leading model-free RL algorithms (SAC, TD3, DDPG, TRPO, PPO) across all tested environments without task-specific hyperparameter tuning.
• The algorithm demonstrates a stable learning process and robust performance, even with varying reward scales.
• Successful application in controlling a wheeled robot highlights its practical deployment potential.

Conclusions:

• DSAC-T represents a significant advancement in reinforcement learning, offering improved Q-value estimation and stable performance.
• The refined algorithm is effective for complex decision-making and control tasks, including real-world robotic applications.
• DSAC-T provides a robust and reliable solution for model-free reinforcement learning challenges.