Learning safe behaviour via justified human preferences and hypothetical queries

Although reinforcement learning is a powerful paradigm for agent sequential decision-making, it cannot be used in its traditional form in most safety-critical environments. Human feedback can enable an agent to learn a good policy while avoiding unsafe states, but at the cost of human time. We present JPAL-HA, a model for safe learning in safety-critical environments that is grounded on two novel ideas: (i) human preferences over a choice of actions are augmented with justifications such as one action is preferred because the other is unsafe; and (ii) we use these justifications to guide the generation of future queries over hypothetical actions to enable the agent to more effectively map out unsafe regions. We evaluate JPAL-HA using modified configurations of the Island Navigation environment from the AI Safety Gridworlds. We empirically show that it significantly improves safety compared to existing approaches for safe learning from human preferences and outperforms other safe human-in-the-loop algorithms in terms of sample efficiency. We also include an analysis of safety/human burden trade-off. Additionally, we evaluate our method with people training a mobile robot, demonstrating that a diverse group of people can complete training in only a few minutes and that hypothetical actions not only further improve safety, but also slightly decrease the training time and keep it to admissible levels.