Cloud Server Replenishment Policy Under Given Demand Satisfaction Rates via Reinforcement Learning With Policy...

Cloud service providers face the challenge of determining optimal server replenishment policies that minimize inventory costs while ensuring the expected demand satisfaction rate. This article addresses a long-term, single-echelon inventory optimization problem tailored to the unique characteristics of cloud, incorporating varying given demand satisfaction rates for different cloud products. Traditional replenishment methods, based on statistical approaches or conventional reinforcement learning (RL) algorithms, are ill-suited due to the uncertainties in user demand retention and the diversity in demand satisfaction requirements. To overcome these limitations, we propose an enhanced replenishment policy under given demand satisfaction rates using RL, combined with our policy imitation. Specifically, we introduce a data-driven safety stock (DSS) model as an expert policy, utilizing Kullback-Leibler (KL) divergence and advantage values for effective action imitation. Based on the results of DSS, we design an efficient reward function and propose an approach for hyperparameter estimation. Together, they enable our method to learn better policies that align with the given demand satisfaction rate. We further design an end-to-end network that integrates demand retention prediction with RL, incorporating a learnable Gaussian kernel to model the lead time effects and local features, and a mean self-attention module for capturing global features. To accelerate training, we also introduce a parallel algorithm. Extensive experiments on real-world data from multiple internet data centers (IDCs), across different time scales and given demand satisfaction rates, demonstrate the superior performance of our method. Additionally, ablation studies and parameter sensitivity analyses verify the efficacy of our policy imitation method and network architecture.