Global-Local Decomposition of Contextual Representations in Meta-Reinforcement Learning

Meta-reinforcement learning (meta-RL) algorithms extract task information from experienced context in order to reason about new tasks, and facilitate rapid adaptation. The quality of these contextual representations (or embeddings) is therefore crucial for a meta-RL agent to make effective decisions in unknown environments. Current methods predominantly assume the existence of a single underlying task, but using a single contextual embedding may not be expressive enough to fully capture the broader distribution of task variations that an agent might encounter. Decomposing that information into different representations can allow them to capture more relevant features in context space while applying additional structure that aids downstream exploitation. In this article, we develop global-local embeddings for contextual meta-RL (GLOBEX), an off-policy contextual meta-RL algorithm that decomposes the contextual representation into separate global and local embeddings. The learning process maximizes information retained by the embeddings and utilizes a mutual information constraint to encourage decoupling. Illustrative examples show that our method effectively adapts by identifying global task dynamics and exploiting temporally local signals. In addition, GLOBEX outperforms existing state-of-the-art meta-RL algorithms on standard MuJoCo benchmarks.