A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) excels at complex tasks but suffers from catastrophic forgetting with continuous data streams.
• Lifelong learning in RL requires methods to integrate new knowledge without degrading past performance.

Purpose of the Study:

• To develop a scalable lifelong reinforcement learning method that dynamically expands network capacity.
• To prevent catastrophic forgetting and interference when learning from streaming information.
• To enable models to generalize and adapt to unseen tasks.

Main Methods:

• Utilized a Dirichlet process mixture to model nonstationary task distributions and cluster task models in a latent space.
• Employed a Chinese restaurant process (CRP) for prior distribution, instantiating new components as needed.
• Applied Bayesian nonparametric framework with expectation maximization (EM) for dynamic model adaptation and domain randomization for robust initialization.

Main Results:

• Demonstrated successful facilitation of scalable lifelong reinforcement learning.
• Showcased superior performance compared to existing methods in robot navigation and locomotion tasks.
• Validated the method's ability to dynamically adapt model complexity without explicit task boundaries.

Conclusions:

• The proposed method effectively addresses catastrophic forgetting in lifelong reinforcement learning.
• The dynamic network expansion and Bayesian nonparametric approach enable robust adaptation to new and unseen tasks.
• The approach shows significant promise for real-world applications requiring continuous learning in robotics.