The effect of attention in cooperative MARL environments with shared rewards

Scalability and coordination remain major challenges in training Multi-Agent Reinforcement Learning (MARL) algorithms. One approach postulates the Centralized Training and Decentralized Execution, which assumes full access to observations from the environment during training but limits agents' reliance on the joint observations within the execution phase. However, this often leads to a rapid increase in input dimensions of the centralized component (critic). Previous studies have suggested using attention mechanisms to enhance scalability and coordination in domains like Treasure Collection and Rover-Tower. This paper aims to complement these findings and offer new insights into the role of attention in MARL, focusing on these two domains on which attention was shown to be beneficial. We show that the impact of attention is very specific and different in the two domains studied. We use manually designed policies to inform our analysis, and explore the challenges concealed in the domains. We argue that the role of attention in the first domain is mainly to provide convenient inductive bias because the local observations of the agents surprisingly contain the same information as the joint observations. In the second domain, the local observations make the exploration challenging due to partial observability in one type of agents and the 'lazy agent' phenomenon. In this case, the role of centralized critic with attention is to mitigate the lazy agent phenomena and partial observability, and the attention itself acts as a simple averaging mechanism.