Kinodynamic Motion Planning With Continuous-Time Q-Learning: An Online, Model-Free, and Safe Navigation Framework

Area of Science:

• Robotics
• Artificial Intelligence
• Control Theory

Background:

• Online kinodynamic motion planning is crucial for autonomous systems.
• Existing methods often struggle with real-time adaptation and optimality.
• Integrating learning-based approaches with sampling-based planners offers potential improvements.

Purpose of the Study:

• To present a novel online kinodynamic motion planning framework, RRT-Q*, that integrates asymptotically optimal Rapidly-exploring Random Tree (RRT*) with continuous-time Q-learning.
• To develop a model-free Q-based advantage function and utilize integral reinforcement learning for online approximation of optimal cost and policy in continuous-time linear systems.
• To ensure stability, asymptotic convergence, and collision-free navigation through Lyapunov-based proofs and novel obstacle handling techniques.

Main Methods:

• Formulation of a model-free Q-based advantage function.
• Application of integral reinforcement learning for tuning laws in online policy approximation.
• Development of a terminal state evaluation procedure for online implementation.
• Introduction of static obstacle augmentation and a local replanning framework based on topological connectedness.

Main Results:

• Rigorous Lyapunov-based proofs demonstrating stability and asymptotic convergence properties.
• Successful online approximation of optimal cost and policy for continuous-time linear systems.
• Demonstrated capability for collision-free navigation through local replanning and obstacle augmentation.
• Validation of the RRT-Q* framework's efficacy via simulations and qualitative comparisons.

Conclusions:

• The RRT-Q* framework provides an effective online solution for kinodynamic motion planning.
• The integration of RRT* and continuous-time Q-learning ensures asymptotic optimality and stability.
• The proposed methods enable robust, collision-free navigation in complex environments.