Dynamics of Q-Learning in Networked Stochastic Games

Area of Science:

• Game Theory
• Computational Economics
• Artificial Intelligence

Background:

• Stochastic games are fundamental to multiagent interactions and reinforcement learning.
• Previous models often simplified interactions to two agents or well-mixed populations, lacking realism.
• Structured populations, where agents interact with neighbors, offer a more realistic scenario.

Purpose of the Study:

• To develop a dynamical model for Q-learning in stochastic games on regular graphs.
• To analyze the influence of population structure and state transition rules on agent dynamics.
• To provide insights into cooperation evolution in social dilemmas within structured populations.

Main Methods:

• Utilized the pair-approximation method to create a novel dynamical model.
• Modeled Q-learning dynamics specifically for agents on regular graphs.
• Validated the model against agent-based simulation results across diverse scenarios.

Main Results:

• The dynamical model accurately predicts Q-learning behavior in structured populations.
• State transition rules can significantly enhance cooperation in social dilemmas under specific conditions.
• Agent degree's impact on cooperation is context-dependent, influenced by transition rules, contrary to prior research.

Conclusions:

• The developed model offers accurate qualitative and quantitative insights into population dynamics in stochastic games.
• Findings highlight the crucial role of state transitions and graph topology in shaping cooperation.
• The study provides a nuanced understanding of how network structure and agent behavior interact to influence collective outcomes.