LLM-Augmented Multi-Agent Reinforcement Learning for Cross-Scenario Knowledge Transfer.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multi-agent reinforcement learning (MARL) faces challenges with low sample efficiency due to extensive trial-and-error interactions.
• Real-world applications of MARL are limited by the high cost of data acquisition and training.
• Accelerating policy convergence and improving transferability are critical for advancing MARL.

Purpose of the Study:

• To propose LoLM-MARL, a novel MARL policy transfer method leveraging large language models (LLMs).
• To enhance sample efficiency and accelerate convergence in multi-agent collaborative tasks.
• To enable effective cross-scenario policy transfer in complex dynamic environments.

Main Methods:

• Fine-tuning pre-trained LLMs using low-rank adaptation (LoRA) to imbue general decision-making knowledge.
• Designing a dynamic prompt construction method that optimizes agent state information for LLMs.
• Implementing Kullback-Leibler (KL) divergence regularization with an annealing strategy to prevent catastrophic forgetting.

Main Results:

• LoLM-MARL achieved up to a 101.4% improvement in average win rate on zero-shot transfer tasks compared to state-of-the-art (SOTA) methods.
• Consistent improvements in generalization performance were observed across six few-shot transfer tasks.
• Convergence speed increased by 4 to 30 times compared to training from scratch.

Conclusions:

• LoLM-MARL offers a new paradigm for efficient policy transfer in MARL by utilizing LLM capabilities.
• The method significantly improves learning efficiency and generalization in complex multi-agent systems.
• Dynamic prompt design and KL regularization are key components for successful LLM-based MARL policy transfer.