Hierarchical meta-reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Meta-Reinforcement Learning (Meta-RL) facilitates rapid adaptation to novel tasks.
• Learning effective policies across diverse, high-dimensional tasks remains a significant challenge.

Purpose of the Study:

• To introduce a novel hierarchical architecture for Meta-RL to enhance policy learning.
• To improve sample efficiency and adaptation capabilities in complex task environments.

Main Methods:

• A three-level hierarchical architecture was proposed: task representation learning, automated discovery of task-agnostic macro-actions, and primitive action learning.
• Macro-actions guide low-level policies, preventing forgetting and enabling efficient state transitions.
• Task-specific components were removed from the state space to ensure macro-action reusability across tasks.
• Independently tailored training schemes were developed to mitigate hierarchical instability.

Main Results:

• The proposed architecture demonstrated improved sample efficiency compared to state-of-the-art methods in MetaWorld experiments.
• Experiments showed a higher success rate in adapting to new tasks.
• The task-agnostic macro-actions proved amenable to re-composition, leading to promising fast adaptation.

Conclusions:

• The novel hierarchical Meta-RL approach effectively addresses challenges in learning performant policies across complex tasks.
• The method significantly enhances sample efficiency and adaptation speed, outperforming existing techniques.
• The task-agnostic macro-action discovery and hierarchical structure offer a promising direction for future Meta-RL research.