Understanding collective behaviors in reinforcement learning evolutionary games via a belief-based formalization

Collective behaviors by self-organization are ubiquitous in nature and human society and extensive efforts have been made to explore the mechanisms behind them. Artificial intelligence (AI) as a rapidly developing field is of great potential for these tasks. By combining reinforcement learning with evolutionary game (RLEG), we numerically discover a rich spectrum of collective behaviors-explosive events, oscillation, and stable states, etc., that are also often observed in the human society. In this work, we aim to provide a theoretical framework to investigate the RLEGs systematically. Specifically, we formalize AI-agents' learning processes in terms of belief switches and behavior modes defined as a series of actions following beliefs. Based on the preliminary results in the time-independent environment, we investigate the stability at the mixed equilibrium points in RLEGs generally, in which agents reside in one of the optimal behavior modes. Moreover, we adopt the maximum entropy principle to infer the composition of agents residing in each mode at a strictly stable point. When the theoretical analysis is applied to the 2×2 game setting, we can explain the uncovered collective behaviors and are able to construct equivalent systems intuitively. Also, the inferred composition of different modes is consistent with simulations. Our work may be helpful to understand the related collective emergence in human society as well as behavioral patterns at the individual level and potentially facilitate human-computer interactions in the future.