A Multi-Agent Continual reinforcement learning framework with multi-Timescale replay and dynamic task classification

This paper proposes an innovative Multi-Agent Continual Reinforcement Learning (MACRL) framework to address the challenges of continual learning in dynamic multi-agent systems. Traditional reinforcement learning suffers from catastrophic forgetting and inefficient cross-task knowledge transfer in non-stationary environments. To overcome these limitations, we introduce two key components: (1) a Multi-Timescale Replay (MTR) buffer, which hierarchically stores experiences across varying timescales to balance new task learning and prior knowledge retention, and (2) a dynamic task classification mechanism that employs an attention-based contextual encoder to measure task similarity and adaptively route policies, thereby minimizing inter-task interference. Experiments on cooperative multi-agent benchmarks (LBF and PP) demonstrate that our framework achieves up to higher average return compared to baselines in sequential task learning, with superior zero-shot generalization performance. Ablation studies further validate the critical roles of MTR and task classification in mitigating catastrophic forgetting. This work provides a scalable solution for collaborative decision-making in complex, evolving environments.