Distributed Policy Space Response Oracles in Two-Player Zero-Sum Games.

Area of Science:

• Artificial Intelligence
• Game Theory
• Reinforcement Learning

Background:

• Policy Space Response Oracle (PSRO) is a population-based algorithm for solving two-player zero-sum games.
• Optimizing policy diversity is critical in PSRO for nontransitive games to prevent exploitation.
• Integrating deep reinforcement learning with PSRO is challenging due to fragmented coordination.

Purpose of the Study:

• To propose a distributed PSRO framework for efficiently solving complex game scenarios.
• To enhance policy diversity and manage optimization costs in PSRO.
• To improve decision-making in game agents through opponent modeling.

Main Methods:

• Developed a distributed PSRO framework with integrated diversity estimation.
• Introduced TOP-K truncation to prioritize high-quality opponents and limit policy pool size.
• Implemented the opponent first (OF) method for enhanced interaction sampling.

Main Results:

• TOP-K truncation effectively enhances diversity while maintaining computational efficiency.
• The distributed training framework integrates diversity optimization without additional overhead.
• Experimental validation on nontransitive models, AlphaStar888, and Google Research Football demonstrated effectiveness.

Conclusions:

• Distributed PSRO with TOP-K truncation and the OF method offers an efficient solution for complex games.
• The proposed methods improve policy diversity, reduce computational costs, and enhance agent decision-making.
• The framework is feasible and efficient for large-scale game environments.