Curriculum-Based Asymmetric Multi-Task Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multi-task reinforcement learning (RL) presents challenges in optimizing performance across diverse tasks.
• Existing curriculum-based methods often suffer from suboptimal training orders and negative transfer.
• Efficiently leveraging prior knowledge and adapting training strategies are crucial for effective multi-task RL.

Purpose of the Study:

• To introduce CAMRL, the first curriculum-based asymmetric multi-task learning (AMTL) algorithm designed for handling multiple RL tasks simultaneously.
• To develop a dynamic training approach that mitigates negative transfer and optimizes performance in AMTL.
• To enable flexible utilization of multi-sourced prior knowledge within the AMTL framework.

Main Methods:

• CAMRL employs a dynamic training mode switching between parallel single-task RL and asymmetric multi-task RL (MTRL) based on performance indicators.
• A composite loss function, incorporating multiple differentiable ranking functions, is optimized using alternating optimization and the Frank-Wolfe algorithm.
• Uncertainty-based automatic hyper-parameter adjustment and continuous revisiting of transfer matrices and network weights are utilized.

Main Results:

• Experiments across diverse benchmarks (Gym-minigrid, Meta-world, Atari, PyBullet, RLBench) demonstrate CAMRL's superior performance.
• CAMRL significantly outperforms both single-task RL algorithms and state-of-the-art MTRL approaches.
• The algorithm effectively predicts subsequent training tasks and adapts its internal representations.

Conclusions:

• CAMRL represents a significant advancement in curriculum-based AMTL for reinforcement learning.
• The proposed dynamic training strategy and composite loss optimization effectively address negative transfer and enhance learning efficiency.
• CAMRL offers a robust and adaptable solution for complex multi-task RL scenarios.