The Most Overestimated Q Value Regularization in High-Dimensional Discrete Action Spaces for Offline Reinforcement...

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Deep reinforcement learning (DRL) is vital for robotic manipulation but hindered by data collection costs and risks.
• Offline reinforcement learning (RL) trains on existing data but suffers from Q-value overestimation in high-dimensional discrete action spaces, impacting stability.
• Out-of-distribution (OOD) actions increase rapidly in these spaces, exacerbating Q-value overestimation.

Purpose of the Study:

• To introduce a novel offline RL algorithm, Most Overestimated Q value Regularization (MQR), designed to mitigate Q-value overestimation.
• To enhance training stability and prevent policy convergence errors in high-dimensional discrete action spaces for robotic manipulation.
• To validate MQR's effectiveness in challenging robotic tasks with diverse environmental conditions.

Main Methods:

• Developed MQR, an offline RL algorithm that specifically penalizes the action with the most overestimated Q-value.
• Implemented a targeted regularization strategy, differing from uniform penalties in existing methods.
• Evaluated MQR on a robotic pushing and grasping task in simulated and real-world settings with varied object arrangements.

Main Results:

• MQR significantly outperformed baseline algorithms in robotic manipulation tasks.
• Achieved a 96.94% clearance rate in simulations and 99.04% in real-world dense configurations.
• Demonstrated high action efficiency and training stability, highlighting MQR's robustness and scalability.

Conclusions:

• MQR effectively mitigates Q-value overestimation in high-dimensional discrete action spaces for offline RL.
• The algorithm shows strong performance, robustness, and adaptability for real-world robotic manipulation.
• MQR holds significant potential for deployment in industrial robotics applications.