Cloud Server Replenishment Policy Under Given Demand Satisfaction Rates via Reinforcement Learning With Policy...

Area of Science:

• Operations Research
• Computer Science
• Artificial Intelligence

Background:

• Cloud providers struggle to balance inventory costs with demand satisfaction rates for diverse products.
• Traditional inventory methods fail due to unpredictable user demand and varied satisfaction requirements.
• Reinforcement learning (RL) is promising but requires adaptation for cloud-specific inventory challenges.

Purpose of the Study:

• To develop an optimized server replenishment policy for cloud environments.
• To improve inventory cost efficiency while meeting specific demand satisfaction rates.
• To address limitations of traditional methods in dynamic cloud settings.

Main Methods:

• Proposed an enhanced replenishment policy using RL combined with policy imitation.
• Introduced a data-driven safety stock (DSS) model as an expert policy, using KL divergence and advantage values.
• Developed an end-to-end network integrating demand retention prediction with RL, featuring a learnable Gaussian kernel and self-attention module.

Main Results:

• The proposed RL method, guided by the DSS expert policy, learned superior replenishment policies aligned with given demand satisfaction rates.
• Experiments on real-world internet data center (IDC) data demonstrated significant performance improvements.
• Ablation studies and sensitivity analyses confirmed the effectiveness of the policy imitation and network architecture.

Conclusions:

• The enhanced RL policy effectively optimizes cloud server replenishment, minimizing costs and maximizing demand satisfaction.
• Policy imitation using a DSS model is a viable strategy for training effective RL agents in complex inventory systems.
• The integrated network architecture and parallel training algorithm enhance performance and efficiency for real-world cloud operations.