Any-step dynamics model improves future predictions for offline reinforcement learning via hierarchical roll-out

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline reinforcement learning optimizes policies using existing datasets, avoiding real-world risks.
• Model-based methods in offline RL enable exploration within data-driven models but struggle with long-horizon predictions.
• Existing methods suffer from compounding errors due to step-by-step bootstrapping predictions.

Purpose of the Study:

• To introduce a novel approach for accurate long-horizon prediction in model-based offline reinforcement learning.
• To mitigate compounding errors inherent in traditional model roll-out techniques.
• To improve model uncertainty estimation and overall performance in offline reinforcement learning tasks.

Main Methods:

• Developed the Any-step Dynamics Model (ADM) to predict future states using variable-length plans.
• Proposed Hierarchical Roll-out (HiRo), which leverages ADM to reduce bootstrapping to direct prediction.
• Utilized ADM's diverse predictions for enhanced model uncertainty estimation compared to ensembles.

Main Results:

• HiRo demonstrates superior long-horizon prediction capabilities over standard step-by-step roll-outs.
• The method provides a more accurate estimation of model uncertainty.
• Offline reinforcement learning using HiRo achieved better performance than state-of-the-art algorithms.

Conclusions:

• HiRo effectively addresses the challenge of long-horizon prediction errors in model-based offline reinforcement learning.
• The proposed method enhances model uncertainty estimation, crucial for robust policy optimization.
• HiRo represents a significant advancement in offline reinforcement learning, offering improved performance and reliability.