HCPI-HRL: Human Causal Perception and Inference-driven Hierarchical Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Hierarchical Reinforcement Learning (HRL) often requires extensive expert knowledge for subgoal definition, limiting its efficiency and adaptability.
• Complex and dynamic environments pose challenges for current HRL agents due to restricted training and adaptability.

Purpose of the Study:

• To develop a novel method, Human Causal Perception and Inference-driven Hierarchical Reinforcement Learning (HCPI-HRL), that infers effective subgoal structures and critical objects using causal relationships.
• To enhance HRL agents' exploration direction and promote the reuse of learned subgoal structures across tasks.
• To overcome the dependency on expert knowledge in HRL for improved training efficiency and adaptability.

Main Methods:

• Proposed HCPI-HRL with a two-level architecture: a meta-controller for assigning subgoals and a Proximal Policy Optimisation (PPO) based lower level for subgoal execution.
• Utilized human-guided causal perception and inference to discover subgoal structures and identify critical objects from dynamic environmental states.
• Incorporated stable causal relationships to guide intrinsic reward generation and agent exploration.

Main Results:

• HCPI-HRL demonstrated superior performance compared to benchmark methods (hierarchical and adjacency PPO) in discrete and continuous control environments.
• The method showed significant improvements in training efficiency, exploration capability, and transferability of learned policies.
• Experiments validated the effectiveness of human-guided causal modeling in inferring subgoal relationships and enhancing agent learning.

Conclusions:

• HCPI-HRL successfully addresses the limitations of traditional HRL by automating subgoal discovery through human causal insights.
• The proposed approach enhances agent capabilities in dynamic environments with sparse rewards, paving the way for more adaptable and efficient HRL agents.
• This research highlights the potential of integrating causal inference with HRL for more sophisticated and autonomous AI systems.