Dynamics of Boltzmann Q learning in two-player two-action games

Area of Science:

• Game theory
• Reinforcement learning
• Computational economics

Background:

• Q learning is a reinforcement learning algorithm used to find optimal strategies in games.
• Boltzmann exploration is a common mechanism for introducing stochasticity in agent decision-making.
• Nash equilibria (NEs) represent stable strategy profiles in game theory where no player can unilaterally improve their outcome.

Purpose of the Study:

• To analyze the convergence properties of Q learning in two-player, two-action games with Boltzmann exploration.
• To characterize the structure of rest points in Q learning dynamics and their relationship to Nash equilibria (NEs).
• To investigate the impact of exploration rates on the stability and existence of these rest points.

Main Methods:

• Mathematical analysis of Q learning dynamics under a Boltzmann exploration mechanism.
• Characterization of the system's rest points for various game structures.
• Sensitivity analysis of rest point structure with respect to exploration noise.

Main Results:

• For any non-zero exploration rate, Q learning dynamics are dissipative, ensuring convergence to rest points.
• These rest points often differ from the game's Nash equilibria (NEs).
• Drastic changes in learning dynamics occur at critical exploration rates for games with multiple NEs.
• Unique, non-NE rest points can emerge and persist within specific exploration rate ranges for games with a single NE.

Conclusions:

• Q learning with Boltzmann exploration does not guarantee convergence to Nash equilibria (NEs) in two-player games.
• The exploration mechanism significantly influences the convergence behavior and the nature of the resulting stable states.
• Understanding these dynamics is crucial for designing effective learning agents in strategic environments.