Continuous-time Q-learning for infinite-horizon discounted cost linear quadratic regulator problems

Area of Science:

• Control Theory
• Reinforcement Learning
• System Identification

Background:

• Traditional Linear Quadratic Regulator (LQR) methods for continuous-time systems often require full or partial knowledge of system dynamics.
• Q-learning is a powerful reinforcement learning technique for unknown dynamical systems, but its application to continuous-time systems is less understood.

Purpose of the Study:

• To develop a Q-learning methodology for solving the discounted Linear Quadratic Regulator (LQR) problem in continuous-time (CT) systems.
• To enable LQR problem solutions for CT systems without prior knowledge of their dynamics.

Main Methods:

• A Q-learning approach is proposed for continuous-time, continuous-state systems.
• A justified parameterization of the Q-function is defined using state, control input, and their derivatives.
• An online Q-learning algorithm is implemented based on this parameterization.

Main Results:

• The presented Q-learning methodology successfully solves the LQR problem for CT systems without requiring system dynamics information.
• Simulation results validate the theoretical framework and the effectiveness of the online learning algorithm.

Conclusions:

• The study demonstrates a viable Q-learning approach for LQR control in unknown CT systems.
• This work extends the applicability of Q-learning to continuous-time control problems, offering a data-driven solution.