Imitation Learning From Suboptimal Demonstrations via Meta-Learning an Action Ranker

A major bottleneck in imitation learning (IL) is the requirement of a large number of expert demonstrations, which can be expensive or even impossible to collect. Learning from a few expert demonstrations in addition to many supplementary potentially suboptimal demonstrations has emerged as a powerful paradigm to address this challenge. However, previous methods often fail to fully utilize the potential of these additional demonstrations by simply discarding nonexpert data. In contrast, our key insight is that suboptimal demonstrations can still be leveraged for policy training if they outperform the current policy. To realize this idea, we propose a novel approach named IL via meta-learning an action ranker (ILMAR), which performs weighted behavior cloning (BC) on a limited set of expert demonstrations along with supplementary demonstrations. It utilizes the functional of the advantage function to selectively integrate knowledge from the supplementary demonstrations. To make more effective use of supplementary demonstrations, we introduce a meta-goal in ILMAR to optimize the functional of the advantage function by explicitly minimizing the Kullback-Leibler (KL) divergence between the current policy and the expert demonstrations. Comprehensive experiments across a wide range of tasks demonstrate that ILMAR significantly outperforms previous methods in handling suboptimal demonstrations. The code is available at https://github.com/F-GOD6/ILMAR.