Value-Regularized Reinforcement Learning for Model Predictive Control of Autonomous Mobile Robots Under Stochastic...

Area of Science:

• Robotics
• Control Systems
• Machine Learning

Background:

• Autonomous mobile robots require robust control for reliable operation.
• Model predictive control (MPC) and reinforcement learning (RL) are key techniques, but face challenges with process disturbances and model mismatch.
• Existing RL-MPC methods often rely on nominal parameters, limiting performance in real-world scenarios.

Purpose of the Study:

• To introduce a novel control strategy, value-regularized RL-MPC (VR-RLMPC), to improve the robustness and performance of autonomous mobile robots.
• To enhance the RL-MPC framework by incorporating isotropic state regularization into the stage cost.
• To evaluate the effectiveness of VR-RLMPC against traditional RL-MPC and other baselines under various conditions, including model mismatch.

Main Methods:

• Introduced isotropic state regularization into the RL-MPC stage cost to create VR-RLMPC.
• Utilized a Lyapunov-drift analysis to establish theoretical performance bounds under specific assumptions.
• Conducted simulations on linear and nonlinear nonholonomic vehicle systems to compare VR-RLMPC with RL-MPC and stochastic temporal-difference baselines.

Main Results:

• VR-RLMPC demonstrated approximately 25% lower settling-step indices compared to RL-MPC across nominal and model-mismatch scenarios.
• The worst-case performance degradation rate for VR-RLMPC was significantly lower (2.8%) than conventional MPC baselines (25-49%) under model mismatch.
• VR-RLMPC showed improved cost performance with increasing task difficulty and outperformed a stochastic temporal-difference baseline in nominal settings.

Conclusions:

• VR-RLMPC offers a significant improvement in control robustness and performance for autonomous mobile robots, particularly under model mismatch.
• The proposed method effectively enhances RL-MPC without increasing computational complexity or online optimization requirements.
• VR-RLMPC represents a promising advancement for reliable autonomous system control in dynamic environments.