Multiagent reinforcement learning with unshared value functions.

Area of Science:

• Artificial Intelligence
• Game Theory
• Multiagent Reinforcement Learning (MARL)

Background:

• Equilibrium-based MARL combines reinforcement learning and game theory.
• Existing algorithms require computationally expensive mixed strategy equilibria and value function sharing, posing privacy and safety concerns.

Purpose of the Study:

• Develop novel and efficient MARL algorithms that do not require agents to share value functions.
• Explore pure strategy equilibrium solution concepts as alternatives to computationally expensive mixed strategy equilibria.

Main Methods:

• Utilized three types of pure strategy profiles: Nash equilibrium, equilibrium-dominating strategy profile, and nonstrict equilibrium-dominating strategy profile.
• Proposed a multistep negotiation process for finding pure strategy equilibria without value function sharing.
• Introduced the Negotiation-based Q-learning (NegoQ) algorithm.

Main Results:

• NegoQ learns equilibrium policies significantly faster than existing MARL algorithms like correlated Q-learning and Nash Q-learning in grid-world games.
• NegoQ demonstrated strong performance in team Markov games, such as pursuit games, rivaling team-task-oriented algorithms.

Conclusions:

• NegoQ offers an efficient and privacy-preserving approach to equilibrium-based MARL.
• The proposed negotiation process effectively finds pure strategy equilibria, broadening the applicability of MARL.