Hierarchical Adversarial Inverse Reinforcement Learning.

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Imitation learning (IL) aims to replicate expert behavior but struggles with complex, long-horizon tasks requiring hierarchical policies.
• Existing hierarchical IL (HIL) methods often fail to capture causal relationships or jointly optimize high-level and low-level policies, leading to suboptimal performance.

Purpose of the Study:

• To develop a novel HIL algorithm that addresses the limitations of current methods.
• To improve the learning of hierarchical policies by explicitly modeling subtask structures and causal relationships.

Main Methods:

• Propose Hierarchical Adversarial Inverse Reinforcement Learning (H-AIRL), extending the state-of-the-art AIRL algorithm.
• Redefine objectives on extended state/action spaces and introduce a directed information term to enhance causality.
• Develop an Expectation-Maximization (EM) adaptation for learning from unannotated demonstrations.

Main Results:

• H-AIRL demonstrates superior performance compared to state-of-the-art HIL baselines on challenging robotic control tasks.
• The proposed directed information term effectively enhances causality between low-level policies and subtasks.
• The EM adaptation enables learning from readily available, unannotated expert demonstrations.

Conclusions:

• H-AIRL offers a significant advancement in hierarchical imitation learning for complex tasks.
• The algorithm's ability to handle unannotated data and improve causal modeling provides practical advantages.
• This work contributes to more effective policy learning in robotics and AI.