Reinforcement Learning-Based Linear Quadratic Regulation of Continuous-Time Systems Using Dynamic Output Feedback.

Area of Science:

• Control Systems Engineering
• Machine Learning
• Adaptive Control

Background:

• Model-free control is crucial for systems where dynamics are unknown.
• Linear Quadratic Regulation (LQR) is a fundamental optimal control problem.
• Output feedback control is practical when full state information is unavailable.

Purpose of the Study:

• To develop a model-free reinforcement learning solution for continuous-time LQR using dynamic output feedback.
• To learn optimal control parameters solely from input-output data, eliminating the need for system models.
• To address limitations of existing output feedback methods, including discrete approximations and static feedback constraints.

Main Methods:

• A state parametrization scheme reconstructs system states from filtered input-output signals.
• Two novel adaptive dynamic programming Bellman equations for output feedback LQR are derived using policy iteration and value iteration (VI).
• The method avoids discrete approximations and handles systems not stabilizable by static output feedback.

Main Results:

• The proposed dynamic output feedback method is immune to exploration bias and does not require a discounted cost function, ensuring stability and optimality.
• The value iteration (VI) method does not necessitate an initially stabilizing policy, unlike prior work.
• Control parameter estimates converge to solutions obtained from the LQR algebraic Riccati equation.

Conclusions:

• The model-free dynamic output feedback approach offers a robust and effective solution for continuous-time LQR.
• The method advances adaptive dynamic programming by obviating discrete approximations and initial policy requirements.
• Simulation studies validate the efficacy and convergence properties of the proposed algorithms.