Heterogeneous Demand-Aware Multi-Agent Communication Based on Role Representation

Efficient communication can help agents to overcome the limitations of partial observations on decision-making and enhance performance in collaborative multi-agent reinforcement learning (MARL). Researchers focus on constructing teammate models or inferring teammate intentions based on local observations to enhance the agent's cognitive ability of global state, which in turn generates customized messages for teammates and improves communication efficiency. However, existing works ignore the positive guidance of messages for generating policy, which weakens collaboration among agents. Moreover, there is variability of intentions among heterogeneous agents in most collaborative scenarios, which may cause erroneous inference. To address these, we propose a heterogeneous demand-aware multi-agent communication (HDMAC) based on a role representation protocol to improve the implicit collaboration of agents. HDMAC dynamically generates agents' role representations, extracts teammate demands from broadcasted tiny-messages and generates customized messages via the correlation between demands and local observation using cross-attention mechanism. To enhance the training efficiency, we draw on the concept of knowledge distillation and introduce a training paradigm by approximating the ideal policy with joint observation based on the maximum return upper bound. Experimental results demonstrate that HDMAC significantly outperforms baseline algorithms in both typical homogeneous and heterogeneous collaborative multi-agent environments.