Dilated memory in hierarchical reinforcement learning for long-horizontal task.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Real-world tasks are often partially observable, violating the Markov property essential for standard reinforcement learning (RL).
• Existing RL methods struggle with long-horizon tasks due to limitations in memory retention, typically only processing adjacent or short historical sequences.
• Processing entire episodes at each step is computationally infeasible for long-horizon tasks.

Purpose of the Study:

• To address memory limitations in partially observable, long-horizon reinforcement learning tasks.
• To develop a hierarchical memory system capable of handling sparse long-term and detailed short-term information.
• To improve training stability and efficiency in hierarchical RL through a novel off-policy correction algorithm.

Main Methods:

• Proposed Dilated Attentive Memory based Hierarchical Reinforcement Learning (DAM-HRL).
• Employed a Transformer at the higher level to capture sparse key time steps (subtask switching points) for long-term memory.
• Utilized an RNN at the lower level for subtask representation and short-term memory retention.
• Introduced a subtask-specific off-policy correction algorithm integrating subtask switching probabilities into importance sampling.

Main Results:

• DAM-HRL significantly extended memory capacity from 50 to over 3000 steps in partially observable conditions.
• The approach demonstrated robustness to the number of steps within individual subtasks.
• Experimental results validated the effectiveness of the hierarchical memory system and the off-policy correction method.

Conclusions:

• DAM-HRL provides an effective and scalable solution for memory challenges in long-horizon, partially observable RL tasks.
• The proposed hierarchical memory structure, combining sparse long-term and detailed short-term memory, is crucial for optimal decision-making.
• The novel off-policy correction method enhances training stability and efficiency in subtask-based RL.