Model-Free λ-Policy Iteration for Discrete-Time Linear Quadratic Regulation

Area of Science:

• Control Theory
• Reinforcement Learning
• Optimization

Background:

• The linear quadratic regulation (LQR) problem is crucial in control systems.
• Traditional methods for solving LQR often require a model or specific initial conditions.
• Iterative methods like policy iteration (PI) and value iteration (VI) are common.

Purpose of the Study:

• Introduce a model-free lambda-policy iteration (λ-PI) algorithm for discrete-time LQR.
• Develop an iterative solution that bypasses the need for a system model.
• Enhance convergence properties and robustness compared to existing methods.

Main Methods:

• Define novel weighted Bellman and composite Bellman operators.
• Formulate λ-PI as a fixed-point iteration using the composite Bellman operator.
• Employ off-policy reinforcement learning for model-free extension.

Main Results:

• The λ-PI algorithm guarantees convergence through contraction and monotonic properties of the composite Bellman operator.
• λ-PI demonstrates superior convergence rates compared to value iteration (VI).
• Off-policy λ-PI variants exhibit robustness against probing noise.

Conclusions:

• The proposed model-free λ-PI is an effective method for solving discrete-time LQR problems.
• The algorithm eliminates the need for an admissible initial policy, unlike traditional PI.
• Simulation results validate the efficacy and robustness of the λ-PI algorithm.