Personalized Federated Actor-Critic Learning for Joint Cost-Comfort Optimization in Energy Communities

Home energy management systems (HEMS) aim to provide intelligent control of the thermal comfort inside smart buildings with the minimum energy cost, while satisfying the energy consumption requests and increasing the use of energy from renewable sources. The capabilities of these intelligent HEMS agents are restricted due to the personalized observability of the environment, resulting in limited knowledge gathering and potentially sub-optimal decisions. Furthermore, several buildings have recently been organized into small energy communities, with the ultimate goal of sharing intelligence between agents in federated learning schemes.In this context, we propose a personalized federated deep reinforcement learning method using Moreau envelopes (pFedMe) for joint energy cost and household comfort optimization in energy communities that consist of multiple smart homes. Specifically, a Twin-Delayed Deep Deterministic Policy Gradient (TD3) actor-critic model is introduced, dynamically observing the state of the smart home environment and suggesting control actions on the operation of the Energy Storage System and on the regulation of the indoor temperature. The TD3 actor-critic model leads to improved policy performance in the continuous control of these systems, mitigating the overestimation bias and improving the training stability of the intelligent agents. The efficiency of the proposed method is verified via simulations based on real data, achieving a beneficial trade-off between the energy cost and the thermal comfort compared to FedAvg and Fedprox baselines. The results show that the proposed pFedMe framework consistently outperforms FedAvg and FedProx in both convergence speed and overall reward, achieving an energy cost reduction of approximately 10% compared to the other schemes, while exhibiting marginal thermal comfort behavior.