Strangeness-driven exploration in multi-agent reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multi-Agent Systems

Background:

• Centralized training with decentralized execution (CTDE) is a common paradigm in multi-agent reinforcement learning (MARL).
• Effective exploration strategies are crucial for improving the performance and stability of MARL algorithms.
• Existing exploration methods can struggle with the stochasticity inherent in MARL environments.

Purpose of the Study:

• To introduce a novel exploration method for CTDE-based MARL algorithms.
• To enhance the stability and performance of MARL by addressing challenges like stochastic transitions.
• To improve the ability of agents to learn effectively in complex, multi-agent environments.

Main Methods:

• A novel exploration method based on the concept of "strangeness" is proposed.
• Strangeness is quantified by the unfamiliarity of agent observations and visited states.
• An exploration bonus derived from strangeness is combined with extrinsic rewards, and a separate action-value function is used to manage the bonus.
• The method is designed to be robust to stochastic transitions in MARL tasks.

Main Results:

• The proposed exploration method significantly improves the stability of CTDE-based MARL algorithms.
• The method demonstrates substantial performance improvements compared to state-of-the-art MARL baselines.
• Evaluations on the StarCraft II micromanagement benchmark show superior performance, highlighting the method's effectiveness.

Conclusions:

• The novel strangeness-based exploration method is effective for CTDE-based MARL.
• The approach enhances learning stability and achieves superior performance on challenging benchmarks.
• This work offers a promising direction for advancing exploration strategies in multi-agent reinforcement learning.