Inverse Value Iteration and Q-Learning: Algorithms, Stability, and Robustness.

Area of Science:

• Control Theory
• Machine Learning
• Reinforcement Learning

Background:

• Linear Quadratic Regulators (LQRs) are fundamental in control system design.
• Learning cost functions from observed behavior is crucial for understanding and replicating control policies.
• Existing inverse reinforcement learning (RL) methods can be computationally intensive.

Purpose of the Study:

• To propose a novel data-driven, model-free inverse Q-learning algorithm.
• To reconstruct the cost function of an agent in continuous-time LQRs using only state-input trajectories.
• To develop a more efficient alternative to existing inverse RL algorithms.

Main Methods:

• Development of a model-based inverse value iteration scheme.
• Introduction of an online model-free inverse Q-learning algorithm.
• Utilizing agent trajectories (states and control inputs) to recover the cost function without system dynamics knowledge.

Main Results:

• The algorithm successfully reconstructs the cost function from demonstrated trajectories.
• The proposed method demonstrates higher efficiency by avoiding repetitive RL computations.
• Guaranteed asymptotic stability, convergence, and robustness of the algorithm.

Conclusions:

• The data-driven model-free inverse Q-learning algorithm is effective for LQRs.
• The approach offers significant efficiency advantages over existing inverse RL techniques.
• The method provides unbiased solutions and does not require initial stabilizing control policies.