DT-HRL: Mastering Long-Sequence Manipulation with Reimagined Hierarchical Reinforcement Learning

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Robotic manipulators in logistics face challenges with multi-step tasks, frequent switching, and long-term dependencies.
• Existing methods struggle with complex, sequential decision-making in dynamic environments.
• Human motor control offers a model for hierarchical task execution.

Purpose of the Study:

• To propose a novel Hierarchical Reinforcement Learning (HRL) framework for robotic manipulators.
• To improve long-term reasoning, generalization, and task execution in logistics.
• To integrate Decision Transformer (DT) capabilities with hierarchical control structures.

Main Methods:

• A multi-task goal-conditioned Decision Transformer (MTGC-DT) framework was developed.
• A high-level policy models the Markov decision process as a sequence modeling task.
• A low-level policy utilizes parameterized action primitives for physical execution.
• Introduced a path-efficiency loss (PEL) correction and a learnable primitive skill library.

Main Results:

• The Decision Transformer-based Hierarchical Reinforcement Learning (DT-HRL) achieved over 10% higher success rate compared to baselines.
• DT-HRL demonstrated over 8% higher average reward in logistics tasks.
• Ablation experiments showed a normalized score increase of over 2%.

Conclusions:

• The proposed DT-HRL framework effectively addresses long-term dependencies and improves generalization in robotic manipulation.
• Integrating Decision Transformers with HRL offers a promising direction for complex task automation in logistics.
• The framework's modular design with parameterized skills enhances reusability and adaptability.