Policy Iteration Q-Learning for Data-Based Two-Player Zero-Sum Game of Linear Discrete-Time Systems

Area of Science:

• Control Theory
• Reinforcement Learning
• Game Theory

Background:

• Linear discrete-time systems present challenges in solving two-player zero-sum game problems.
• Traditional methods require complete system dynamics and solving the discrete-time game algebraic Riccati equation (DTGARE).

Purpose of the Study:

• To develop a data-based algorithm for solving zero-sum games in linear discrete-time systems.
• To avoid the need for complete system dynamics and the DTGARE.

Main Methods:

• Introduction of the Q-function to bypass DTGARE.
• Development of a data-based policy iteration Q-learning (PIQL) algorithm using real system data.
• Proof of PIQL algorithm's equivalence to Newton's method using Fréchet derivative and guarantee of convergence via Kantorovich's theorem.
• Implementation using an off-policy learning scheme.

Main Results:

• The PIQL algorithm successfully learns the optimal Q-function from real system data.
• The algorithm's convergence is theoretically guaranteed.
• Simulation studies validate the efficiency of the data-based PIQL method.

Conclusions:

• The data-based PIQL algorithm offers an effective approach to solving zero-sum games for linear discrete-time systems.
• This method eliminates the requirement for system models and DTGARE solutions, enabling practical application.
• The findings demonstrate a significant advancement in data-driven control and learning for game-theoretic problems.