WToE: Learning When to Explore in Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Reinforcement Learning
• Robotics

Background:

• Existing multiagent exploration methods are insufficient for nonstationary environments caused by agent interactions.
• Nonstationarity in multiagent systems arises from dynamic changes induced by agent behaviors.

Purpose of the Study:

• To propose a novel exploration method, When to Explore (WToE), designed to address nonstationarity in multiagent environments.
• To develop an adaptive exploration mechanism that effectively handles environmental changes due to agent interactions.

Main Methods:

• Introduced a graphical model with a latent variable to capture step-level environmental changes from interactions.
• Utilized a supervised variational auto-encoder (VAE) framework to infer a short-term policy from historical data.
• Agents explore when the short-term inferred policy diverges from the current actor policy, adapting to nonstationarity.

Main Results:

• The proposed WToE method demonstrates theoretical convergence guarantees for the Q-value function.
• Experimental validation in grid, multiagent particle, and MAgent environments showed WToE's superiority over baselines.
• WToE outperformed established methods like MAEXQ, NoisyNets, EITI, and PR2.

Conclusions:

• WToE is a simple yet effective variational exploration method for nonstationary multiagent environments.
• The interaction-oriented adaptive mechanism allows WToE to successfully manage environmental dynamics.
• WToE offers a significant advancement in multiagent exploration, providing superior performance and theoretical guarantees.