A Decentralized Actor-Critic Algorithm With Entropy Regularization and Its Finite-Time Analysis.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Decentralized actor-critic (AC) methods are dominant in multiagent reinforcement learning (MARL).
• Existing decentralized AC algorithms struggle to simultaneously achieve exploration efficiency, sample efficiency, and communication efficiency.
• There is a need for advanced MARL algorithms that overcome these limitations.

Purpose of the Study:

• To develop a novel decentralized multiagent AC algorithm that improves exploration efficiency.
• To provide theoretical guarantees for the algorithm's sample and communication complexities.
• To evaluate the algorithm's performance on benchmark reinforcement learning tasks.

Main Methods:

• Incorporation of entropy regularization into a decentralized multiagent AC framework.
• Theoretical analysis to derive sample complexity as O(ε⁻²ln ε⁻¹) and communication complexity as O(ε⁻¹ln ε⁻¹).
• Empirical evaluation on various reinforcement learning tasks.

Main Results:

• The proposed multi-agent AC algorithm with entropy regularization (MACE) demonstrates enhanced exploration efficiency.
• MACE achieves theoretical sample and communication complexities matching current state-of-the-art.
• Experimental results confirm MACE's superior performance compared to existing decentralized AC algorithms.

Conclusions:

• MACE effectively addresses the simultaneous challenges of exploration, sample, and communication efficiency in decentralized MARL.
• The algorithm offers a promising advancement for complex multiagent systems.
• MACE provides a theoretically sound and empirically validated solution for MARL problems.