Value Iteration for Stochastic LQR With Convergence Guarantees.

Area of Science:

• Control Theory
• Machine Learning
• Stochastic Systems

Background:

• Linear Quadratic Regulator (LQR) problems are fundamental in control theory.
• Systems often encounter noise with unknown statistical properties, complicating control design.
• Model-free approaches are desirable when system dynamics are not fully known.

Purpose of the Study:

• To develop a model-free (MF) algorithm for the discounted stochastic linear quadratic regulator (LQR) problem.
• To address systems with additive noise of unknown mean.
• To enable learning of optimal control policies and discount factors directly from data.

Main Methods:

• A completely model-free (MF) value iteration (VI) algorithm is proposed.
• The algorithm utilizes off-line system trajectories for policy learning.
• A separate MF algorithm is developed for learning a feasible discount factor.

Main Results:

• The MF VI algorithm converges to a neighborhood of the optimal control policy with high probability.
• The proposed methods are demonstrated through illustrative examples.
• Feasible discount factors can be learned using the developed MF approach.

Conclusions:

• Model-free value iteration provides an effective solution for discounted stochastic LQR problems with unknown noise characteristics.
• The developed algorithms offer practical approaches for learning control policies and discount factors in data-driven scenarios.
• The study highlights the applicability of MF techniques in complex control system design.