Enhancing Value Decomposition With Target Transformation in Cooperative Multi-Agent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Cooperative multi-agent reinforcement learning (MARL) is crucial for coordinating intelligent machines.
• Dominant MARL methods using monotonic value decomposition offer scalability but limit representable joint action-values.
• Existing solutions for non-monotonic targets can be unstable with stochastic returns, favoring suboptimal trajectories.

Purpose of the Study:

• To develop a novel approach for cooperative MARL that overcomes limitations of monotonic value decomposition.
• To address the fragility of learning targets in stochastic environments.
• To improve the performance and stability of MARL agents.

Main Methods:

• Proposed Target Transformation to map non-monotonic and stochastic learning targets to a monotonic-representable surrogate.
• Developed Uncertainty-aware Target Transformation (UT2) with value-based and policy-based instantiations.
• Integrated an uncertainty estimator with a best-individual coordination envelope.

Main Results:

• UT2 demonstrated improved performance and stability across diverse cooperative MARL benchmarks.
• The method showed significant gains with increasing non-monotonicity and stochasticity in returns.
• UT2 successfully preserves the optimal joint action while handling complex learning targets.

Conclusions:

• UT2 offers a robust solution for cooperative MARL, particularly in environments with non-monotonic and stochastic reward structures.
• The approach enhances the reliability and effectiveness of multi-agent coordination.
• This work advances the field of MARL by providing a more stable and performant learning framework.