Constraining an Unconstrained Multi-agent Policy with offline data.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Real-world multi-agent systems require decision-making under constraints (e.g., safety, economics).
• Constrained Multi-Agent Reinforcement Learning (CMARL) addresses this but often relies on online training, which is frequently infeasible.
• New constraints may arise after initial policy training, necessitating adaptation.

Purpose of the Study:

• To propose a method (CUTMAP) for constraining unconstrained multi-agent policies using offline data.
• To enable CMARL without online interaction and facilitate adaptation to new constraints.
• To address the challenges of offline learning, such as distribution shift.

Main Methods:

• Developed a scalable optimization objective within multi-agent maximum entropy reinforcement learning.
• Estimated a decomposable Q-function using an unconstrained prior policy and offline cost signals.
• Incorporated a conservative loss term to mitigate offline distribution shift.

Main Results:

• CUTMAP effectively constrains unconstrained policies using only offline data.
• The method can adapt to new constraints by reusing the prior policy without re-training.
• Demonstrated superior performance across various cooperative multi-agent benchmarks (StarCraft, particle games, robot control).

Conclusions:

• CUTMAP offers a practical solution for CMARL by leveraging offline data.
• It reduces the need for expensive online interactions and simulator development.
• Facilitates the deployment of MARL in real-world applications with evolving constraints.