Online Minimax Q Network Learning for Two-Player Zero-Sum Markov Games.

Area of Science:

• Game Theory
• Artificial Intelligence
• Reinforcement Learning

Background:

• The Nash equilibrium is a key concept in game theory, representing a state of minimal exploitability between opponents.
• Two-player zero-sum Markov games (TZMGs) present complex strategic decision-making challenges.

Purpose of the Study:

• To develop and validate an online learning algorithm for finding Nash equilibrium policies in TZMGs.
• To integrate deep reinforcement learning (DRL) with generalized policy iteration (GPI) for solving TZMGs.

Main Methods:

• Formulated the TZMG problem as a Bellman minimax equation.
• Applied generalized policy iteration (GPI) with neural networks for Q-function approximation.
• Developed an online minimax Q-network learning algorithm incorporating experience replay, dueling networks, and double Q-learning.

Main Results:

• Successfully combined DRL techniques with GPI to determine TZMG Nash equilibrium.
• Proved the convergence of the online learning algorithm, offering insights for single-agent problems.
• Experimental validation demonstrated the algorithm's effectiveness on various TZMG examples.

Conclusions:

• The proposed DRL-based approach offers a novel and effective method for solving TZMGs.
• The convergence proof extends theoretical understanding to both multi-agent and single-agent reinforcement learning.
• This work advances the application of AI in strategic decision-making and game theory.