Structured goal-guided reinforcement learning with large language models

Real-world decision-making tasks typically occur in complex and open environments, posing significant challenges to agents' exploration efficiency and long-horizon planning capabilities. Recent LLM-enhanced RL methods leverage large language models (LLMs) to guide reinforcement learning (RL) agents toward efficient exploration but typically incur high costs and suffer from instability due to a semantic mismatch between goal guidance and policy learning. In this paper, we introduce Structured Goal-guided Reinforcement Learning (SGRL), a novel framework that integrates a Structured Goal Planner and a Goal-Conditioned Action Pruner into a standard RL pipeline to guide exploration efficiently while preserving policy flexibility. Extensive experiments on Crafter, Craftax-Classic, and two reward-modified variants of Craftax-Classic demonstrate that SGRL achieves superior performance on challenging tasks requiring long-horizon exploration. The code is available at https://github.com/jiejieqi/SGRL.