TVDO: Tchebycheff Value-Decomposition Optimization for Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Cooperative multiagent reinforcement learning (MARL) often uses centralized training with decentralized execution (CTDE).
• A key challenge in CTDE is the inconsistency between jointly trained policies and individually executed actions.

Purpose of the Study:

• To propose a novel method, factorized Tchebycheff value-decomposition optimization (TVDO), to resolve policy inconsistency in MARL.
• To ensure consistency between global and individual optimal action-value functions in CTDE.

Main Methods:

• Formulation of a nonlinear Tchebycheff aggregation function inspired by multiobjective optimization (MOO).
• Theoretical proof that the factorized value decomposition with Tchebycheff aggregation satisfies individual-global-max (IGM) sufficiency and necessity.
• Empirical verification in the climb and penalty game and evaluation on the StarCraft multiagent challenge (SMAC) benchmark.

Main Results:

• TVDO precisely expresses global-to-individual value decomposition with guaranteed policy consistency.
• TVDO demonstrates significant performance superiority over state-of-the-art (SOTA) MARL baselines in empirical evaluations.
• The method effectively constrains the upper bound of individual action-value bias to achieve global optimum.

Conclusions:

• TVDO effectively overcomes the inconsistency challenge in CTDE for MARL.
• The proposed method guarantees policy consistency and achieves superior performance in complex MARL environments.
• TVDO offers a promising approach for advancing cooperative MARL research.