Deterministic limit of temporal difference reinforcement learning for stochastic games.

Area of Science:

• Artificial Intelligence
• Game Theory
• Statistical Physics

Background:

• Multiagent reinforcement learning (MARL) analysis often overlooks environmental dynamics in favor of single-state games.
• Existing methods lack a universal approach to derive deterministic equations for MARL in multistate environments.

Purpose of the Study:

• To present a methodological extension for deriving the deterministic limit of temporal difference (TD) learning algorithms.
• To enable MARL analysis in more realistic multistate environments.

Main Methods:

• Separating interaction and adaptation timescales.
• Deriving deterministic equations for a general class of TD learning algorithms.
• Applying the method to Q learning, SARSA learning, and actor-critic learning.

Main Results:

• Demonstrated the method's potential on two multiagent, multistate environments.
• Revealed diverse dynamical regimes including fixed points, limit cycles, and deterministic chaos.
• Provided a universal method for analyzing TD learning in multistate settings.

Conclusions:

• The proposed method offers a way to analyze complex MARL dynamics in realistic environments.
• This work bridges the gap between theoretical MARL and practical applications with dynamic environments.