Decentralized multi-agent reinforcement learning based on best-response policies.

Area of Science:

• Robotics
• Artificial Intelligence
• Multi-Agent Systems

Background:

• Multi-agent systems involve multiple agents interacting in a shared environment.
• Advances in single-agent reinforcement learning have spurred interest in multi-agent reinforcement learning (MARL).
• Centralized learning schemes are common but less practical for real-world, decentralized robot deployments.

Purpose of the Study:

• To propose a decentralized actor-critic (AC) approach for cooperative MARL in sparsely rewarded domains.
• To enable individual robot learning and deployment without centralized control.
• To address challenges in coordinating multiple agents with limited feedback.

Main Methods:

• Developed a novel actor-critic (AC) approach for cooperative MARL.
• Decoupled the MARL problem into distributed agents modeling others as responsive.
• Implemented two critics per agent: one for joint task reward, one for agent-specific costs.
• Utilized Stackelberg game models (game against nature, dyadic game) for decentralized execution and training.

Main Results:

• Evaluated the novel AC approach in a sparsely rewarded simulated multi-agent environment.
• The proposed method demonstrated superior performance compared to state-of-the-art MARL learners.
• The decentralized scheme allows for effective learning and execution on individual robots.

Conclusions:

• The novel decentralized AC approach is effective for cooperative MARL in sparse reward settings.
• The dual-critic system successfully balances joint task optimization and agent-specific cost reduction.
• Future research can build upon this framework for more complex multi-agent coordination challenges.