Massive Coordination of Distributed Energy Resources in VPP: A Mean Field RL-Based Bi-Level Optimization Approach

The coordination of distributed energy resources (DERs) within virtual power plants (VPPs) is expected to generate significant economic benefits and enhance the operational stability of modern power systems. However, achieving massive coordination of heterogeneous and uncertain DERs remains a challenge in current research. To address this issue, this article proposes a novel bi-level optimization approach based on mean-field reinforcement learning (MFRL) to enable the coordination of massive DERs in VPPs. The problem is decomposed into multiple subproblems: the upper-level subproblem models power dispatch among integrated energy systems (IESs) in response to coordinated demand, while a series of lower-level subproblems determine the operational schemes of DERs within individual IESs. Considering the large decision space, an MFRL algorithm with fast Shapley credit allocation is developed to efficiently solve the upper-level optimization. Meanwhile, the lower-level subproblems are formulated as small-scale mixed-integer linear programming (MILP) problems, addressing the difficulties caused by IES heterogeneity in applying mean-field approximation. Simulation results show that the proposed approach significantly improves convergence speed and reduces the global cost of VPP operation, especially in massive-scale scenarios. In test scenarios ranging from 10 to 500 agents, the proposed bi-level optimization approach improves the objective by 4.8%-26.6%, compared to the advanced baseline method.