Adjacency Constraint for Efficient Hierarchical Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Goal-conditioned Hierarchical Reinforcement Learning (HRL) offers scalability for reinforcement learning (RL).
• Training inefficiency arises from large high-level action (goal) spaces in HRL.
• This difficulty impacts both high-level subgoal generation and low-level policy learning.

Purpose of the Study:

• To alleviate training inefficiency in goal-conditioned HRL.
• To improve performance in complex control tasks by addressing large goal spaces.

Main Methods:

• Introducing an adjacency constraint to restrict the high-level action space to a k-step adjacent region.
• Theoretically proving policy preservation in deterministic Markov Decision Processes (MDPs).
• Demonstrating bounded suboptimality in stochastic MDPs.
• Implementing the constraint via an adjacency network for subgoal discrimination.

Main Results:

• The adjacency constraint significantly boosts performance in state-of-the-art goal-conditioned HRL approaches.
• Experimental validation on discrete and continuous control tasks, including robot locomotion and manipulation.
• Theoretical guarantees of policy optimality or bounded suboptimality.

Conclusions:

• The adjacency constraint is an effective method for improving HRL training efficiency and performance.
• This approach successfully addresses the challenges posed by large goal spaces in HRL.
• The findings have implications for advancing RL in complex robotic applications.