Output Feedback Q-Learning Control for the Discrete-Time Linear Quadratic Regulator Problem.

Area of Science:

• Control Systems Engineering
• Machine Learning
• Optimization

Background:

• Approximate dynamic programming (ADP) and reinforcement learning (RL) are key for optimal control.
• Existing methods often require full-state feedback, which is impractical in many applications.
• Output feedback methods offer a more feasible alternative by relaxing this requirement.

Purpose of the Study:

• To develop a novel output feedback Q-learning approach for discrete-time linear quadratic regulation (LQR).
• To design controllers online without prior knowledge of system dynamics.
• To address limitations of existing methods, particularly the need for full-state feedback.

Main Methods:

• Developed a new Q-function representation using input-output data for LQR.
• Designed output feedback LQR controllers based on this representation.
• Implemented two iterative Q-learning algorithms: policy iteration and value iteration.

Main Results:

• The proposed scheme operates online and does not require system dynamics information.
• It avoids excitation noise bias, eliminating the need for discounted cost functions and ensuring closed-loop stability.
• Algorithms demonstrated convergence to the solution of the LQR Riccati equation.

Conclusions:

• The novel output feedback Q-learning approach effectively solves the LQR problem for discrete-time systems.
• The method is practical for real-world applications due to its online nature and relaxed feedback requirements.
• Simulation studies validate the efficacy and stability of the proposed control scheme.