Accelerating Multiagent Reinforcement Learning by Equilibrium Transfer.

Area of Science:

• Artificial Intelligence
• Game Theory
• Machine Learning

Background:

• Equilibrium-based multiagent reinforcement learning (MARL) uses game theory concepts but struggles with scalability due to computationally intensive equilibrium calculations.
• Existing algorithms face challenges in efficiently computing equilibria, such as Nash equilibria, which are often PPAD-hard.

Purpose of the Study:

• To address the scalability limitations of equilibrium-based MARL algorithms.
• To introduce a novel method for accelerating MARL by leveraging the observation that equilibria in successive game states are often similar.

Main Methods:

• Proposed an equilibrium transfer framework for MARL, reusing previously computed equilibria when agents have minimal incentive to deviate.
• Introduced transfer loss and transfer conditions to the MARL framework.
• Theoretically proved convergence to an equilibrium policy under specific assumptions despite transfer loss.

Main Results:

• Demonstrated significant acceleration of equilibrium-based MARL, with up to 96.7% reduction in learning time.
• Achieved higher average rewards compared to algorithms without equilibrium transfer.
• Showcased significantly improved scalability with increasing state/action spaces and agent numbers.

Conclusions:

• Equilibrium transfer is an effective strategy for accelerating and enhancing the scalability of equilibrium-based MARL.
• The proposed framework offers a practical solution to the computational bottlenecks in MARL.