Improving Sample Efficiency of Reinforcement Learning With Background Knowledge From Large Language Models.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Reinforcement learning (RL) faces challenges with low sample efficiency.
• Current methods using large language models (LLMs) for RL guidance lack generalizability across tasks.
• A need exists for reusable environmental knowledge to accelerate RL policy learning.

Purpose of the Study:

• To develop a framework that leverages LLMs to extract generalizable background knowledge of an environment.
• To represent this knowledge as potential functions for effective reward shaping in RL.
• To enhance the sample efficiency of various downstream RL tasks through a one-time knowledge representation.

Main Methods:

• Grounding LLMs with pre-collected environmental experiences.
• Prompting LLMs to delineate background knowledge using methods like code generation, preference annotation, and goal assignment.
• Representing extracted knowledge as potential functions for potential-based reward shaping.

Main Results:

• Demonstrated significant improvements in sample efficiency across multiple RL tasks.
• Validated the framework's effectiveness in the Minigrid and Crafter domains.
• Showcased the generalizability of the extracted environmental knowledge for diverse downstream tasks.

Conclusions:

• The proposed framework effectively utilizes LLMs to capture and represent general environmental knowledge.
• This approach significantly boosts sample efficiency in reinforcement learning.
• The method offers a promising direction for creating more adaptable and efficient RL agents.