LLM-Augmented Multi-Agent Reinforcement Learning for Cross-Scenario Knowledge Transfer

Multi-agent reinforcement learning (MARL) relies on trial-and-error interactions to update policies. However, trial-and-error learning typically requires extensive interactions to achieve satisfactory performance, resulting in low sample efficiency, which limits its application in the real world. To reduce the trial-and-error costs of MARL and accelerate the convergence of multi-agent collaborative policies, we propose a MARL policy transfer method named LoLM-MARL, based on fine-tuning large language models (LLMs). First, leveraging the general world knowledge and reasoning capabilities of LLMs, low-rank adaptation (LoRA) is employed to fine-tune the pre-trained model on source tasks, thereby providing general decision-making knowledge for cross-scenario policy transfer. Second, a dynamic prompt construction method for LLMs is designed. By dynamically eliminating the state information of ineffective agents from the prompts, the method provides denser observation data for the large language model, thereby enhancing its policy representation capability in specific complex collaborative scenarios. Meanwhile, the dynamic prompt design concept enriches the training sub-scenarios for the algorithm, thereby laying the foundation for the model to learn more general decision-making knowledge. Finally, a Kullback-Leibler (KL) divergence regularization method based on an annealing strategy is constructed to ensure consistency between the policy distributions of the fine-tuned model and the pre-trained model, effectively mitigating the catastrophic forgetting problem during the fine-tuning process of the pre-trained model. Experimental results show that in zero-shot transfer tasks, LoLM-MARL achieves a maximum improvement of 101.4% in average win rate compared to existing state-of-the-art (SOTA) methods. In six few-shot transfer tasks, our method consistently achieves better generalization performance than traditional SOTA methods, and improves the convergence speed by 4 to 30 times compared to the training-from-scratch approach, providing a new solution paradigm for efficient policy transfer in complex dynamic environments.