Optimal Power Management Based on Q-Learning and Neuro-Dynamic Programming for Plug-in Hybrid Electric Vehicles

Energy optimization for plug-in hybrid electric vehicles (PHEVs) is a challenging problem due to the system complexity and many physical and operational constraints in PHEVs. In this paper, we present a Q-learning-based in-vehicle learning system that is free of physical models and can robustly converge to an optimal energy control solution. The proposed machine learning algorithms combine neuro-dynamic programming (NDP) with future trip information to effectively estimate the expected future energy cost (expected cost-to-go) for a given vehicle state and control actions. The convergences of these learning algorithms were demonstrated on both fixed and randomly selected drive cycles. Based on the characteristics of these learning algorithms, we propose a two-stage deployment solution for PHEV power management applications. Furthermore, we introduce a new initialization strategy, which combines the optimal learning with a properly selected penalty function. This initialization scheme can reduce the learning convergence time by 70%, which is a significant improvement for in-vehicle implementation efficiency. Finally, we develop a neural network (NN) for predicting battery state-of-charge (SoC), rendering the proposed power management controller completely free of physical models.