Game-Theoretic Inverse Reinforcement Learning: A Differential Pontryagin's Maximum Principle Approach

Area of Science:

• Game Theory
• Reinforcement Learning
• Control Theory

Background:

• Learning system dynamics and cost functions is crucial for multistage games.
• Existing methods often rely on probabilistic or residual minimization approaches.
• A deterministic framework offers potential advantages in certain applications.

Purpose of the Study:

• To propose a novel game-theoretic inverse reinforcement learning (GT-IRL) framework.
• To learn parameters of both the dynamic system and individual cost functions.
• To address multistage games using demonstrated trajectories in a deterministic setting.

Main Methods:

• Developed a GT-IRL framework based on differentiating Pontryagin's maximum principle (PMP) equations for open-loop Nash equilibrium (OLNE).
• Established equivalence between differentiated PMP equations for general multistage games and PMP equations for affine-quadratic games.
• Utilized explicit recursions for solving the derived equations in both multi-player nonzero-sum and two-player zero-sum games.

Main Results:

• The proposed framework successfully learns system dynamics and cost function parameters from demonstrated trajectories.
• Demonstrated equivalence for multi-player nonzero-sum games, enabling explicit solutions.
• Extended the methodology to two-player zero-sum games, showing similar tractability.

Conclusions:

• The GT-IRL framework provides an effective deterministic method for parameter identification in multistage games.
• The approach offers a viable alternative to probabilistic and residual minimization techniques.
• Simulation results validate the algorithm's effectiveness in learning game parameters.