Policy Iteration-Based Learning Design for Linear Continuous-Time Systems Under Initial Stabilizing OPFB Policy.

Area of Science:

• Reinforcement Learning
• Control Theory
• Continuous-Time Systems

Background:

• Policy iteration (PI) is valuable for learning decision laws in unknown environments.
• Existing PI methods for output-feedback (OPFB) continuous-time systems require an initial stabilizing full state-feedback (FSFB) policy, violating the OPFB principle.
• This limitation hinders the direct application of PI in scenarios where only output feedback is available.

Purpose of the Study:

• To establish policy iteration (PI) under an initial stabilizing output-feedback (OPFB) policy for continuous-time systems.
• To address the violation of the OPFB principle in existing PI-based control methods.
• To develop an efficient PI algorithm that approximates optimal control using only OPFB information.

Main Methods:

• An off-policy Bellman equation is utilized to transform any OPFB policy into an FSFB policy.
• The traditional PI algorithm is revised by incorporating an additional iteration based on this transformation.
• Theoretical analysis and a case study are employed to demonstrate the method's effectiveness.

Main Results:

• The proposed method successfully establishes policy iteration (PI) under an initial stabilizing OPFB policy.
• The transformation property of the off-policy Bellman equation enables the use of OPFB policies.
• The revised PI algorithm efficiently approximates the optimal control law under OPFB constraints.

Conclusions:

• The developed PI method effectively overcomes the reliance on initial FSFB policies for continuous-time systems.
• This work provides a theoretical foundation and practical demonstration for PI in OPFB settings.
• The proposed approach enhances the applicability of reinforcement learning for control in real-world systems with limited feedback.