Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Soft Actor-Critic (SAC) is a leading off-policy reinforcement learning algorithm.
• SAC maximizes expected return and policy entropy for state-of-the-art performance on continuous control tasks.
• Standard SAC samples uniformly from an experience replay buffer for policy and value network updates.

Purpose of the Study:

• To improve the performance and sample efficiency of the Soft Actor-Critic algorithm.
• To introduce modifications that enhance the learning process of SAC.
• To develop a more stable and effective reinforcement learning approach for continuous control.

Main Methods:

• Proposed an improved SAC (ISAC) algorithm.
• Introduced a novel prioritization scheme for selecting superior samples from the experience replay buffer.
• Integrated prioritized off-policy data with the latest on-policy data for network training.

Main Results:

• ISAC demonstrated superior performance compared to vanilla SAC and other variants.
• The proposed approach showed increased stability during training.
• ISAC achieved higher sample efficiency in continuous control tasks within MuJoCo environments.

Conclusions:

• The modifications in ISAC significantly boost SAC's performance and sample efficiency.
• ISAC offers a more stable and effective reinforcement learning solution.
• The findings suggest ISAC is a promising advancement for continuous control applications.