Exploring the limits of hierarchical world models in reinforcement learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Hierarchical Model-Based Reinforcement Learning (HMBRL) combines model-based and hierarchical approaches for enhanced sample efficiency and abstraction.
• Current HMBRL frameworks face complexities hindering general principles extraction and adaptation.
• Progress in HMBRL is impeded by challenges in understanding and applying diverse use cases.

Purpose of the Study:

• Introduce and evaluate a novel HMBRL framework.
• Explore static and environment-agnostic temporal abstraction for concurrent training.
• Address challenges in abstract model exploitation within hierarchical structures.

Main Methods:

• Constructed hierarchical world models with varying temporal abstraction levels.
• Trained a stack of agents communicating goals top-down.
• Focused on static, environment-agnostic temporal abstraction for low-dimensional abstract actions.
• Evaluated the framework's ability to facilitate decision-making across abstraction levels.

Main Results:

• The proposed HMBRL approach facilitated decision-making across two abstraction levels.
• Static temporal abstraction allowed concurrent training of models and agents.
• The framework did not outperform traditional methods in final episode returns.
• Model exploitation on the abstract level of the world model stack emerged as a key challenge.

Conclusions:

• The novel HMBRL framework demonstrates potential for multi-level decision-making.
• Static temporal abstraction offers advantages in concurrent training and action space dimensionality.
• Further research is needed to address challenges in abstract model exploitation for performance enhancement.
• This work contributes to refining HMBRL methodologies and understanding its limitations.