Multi-Agent Deep Reinforcement Learning for Multi-Echelon Inventory Management

Area of Science:

• Operations Research
• Artificial Intelligence
• Supply Chain Management

Background:

• Decentralized multi-echelon inventory management presents complex challenges.
• The bullwhip effect significantly distorts demand information across supply chains.
• Existing heuristic and single-agent reinforcement learning methods have limitations.

Purpose of the Study:

• To apply a multi-agent deep reinforcement learning algorithm (HAPPO) to decentralized multi-echelon inventory management.
• To evaluate HAPPO's effectiveness in reducing overall costs and mitigating the bullwhip effect.
• To investigate the impact of information-sharing mechanisms within MADRL on supply chain performance.

Main Methods:

• Heterogeneous-Agent Proximal Policy Optimization (HAPPO), a multi-agent deep reinforcement learning (MADRL) algorithm.
• Application to serial and network supply chain structures.
• Comparison with single-agent deep reinforcement learning and heuristic policies.

Main Results:

• HAPPO-derived policies achieved lower overall costs than single-agent and heuristic policies.
• HAPPO demonstrated a reduced bullwhip effect compared to non-information-sharing single-agent methods.
• A combined cost objective (individual and system) for actors yielded better results than purely system-focused or self-interested objectives.

Conclusions:

• MADRL, specifically HAPPO, offers a powerful approach for optimizing complex, decentralized supply chains.
• Upfront information sharing and coordinated action during training are crucial for MADRL success in supply chains.
• A balanced actor objective (individual and system costs) is key for optimal policy performance in MADRL-based supply chain management.