Trajectory-Constrained Human-Guided Reinforcement Learning for Autonomous Driving

In this article, we propose trajectory-constrained human-guided reinforcement learning (TCHug-RL), a new framework that 1) produces guidance at the full-trajectory level to suit the planning horizon of autonomous driving and 2) provides a formal criterion for deciding when and how to inject human input into the policy-update loop. First, we propose a trajectory-level similarity of human guidance, which is defined such that the satisfaction of trajectory-level similarity implies the satisfaction of single-step similarity. Consequently, trajectory-level similarity is strictly stronger than the single-step one. Then, the trajectory-level similarity is modeled as a constraint to guide the subsequent policy optimization. In this way, not only is a clear criterion for human guidance provided, but the influence of such guidance on the policy is also transparent and interpretable. Finally, we leverage the Lagrangian method to solve the constrained optimization problem and provide a strong duality analysis for it in the unparameterized policy space. Furthermore, a practical algorithm implementation of TCHug-RL and experiments are provided, demonstrating that TCHug-RL effectively leverages human guidance and achieves improvements of 22.1% in learning efficiency and 18.7% in overall performance in autonomous driving tasks, compared to the state-of-the-art methods.