Intelligent Trainer for Dyna-Style Model-Based Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-based reinforcement learning (MBRL) offers a solution to high sampling costs in traditional RL by using system dynamics models for synthetic data generation.
• However, MBRL's effectiveness is hindered by the complex, manual tuning required for jointly optimizing policies, dynamics models, and data sampling strategies, leading to prohibitive costs.

Purpose of the Study:

• To address the challenges of manual hyperparameter tuning and high costs in MBRL.
• To introduce a novel "reinforcement on reinforcement" (RoR) architecture for automated hyperparameter optimization in MBRL.

Main Methods:

• Proposed a "reinforcement on reinforcement" (RoR) architecture, decoupling MBRL into two RL layers: an inner "training process environment" (TPE) and an outer "intelligent trainer."
• Developed and optimized two trainer designs: a unihead trainer and a multihead trainer.
• Evaluated the RoR framework on five OpenAI Gym tasks, comparing it against three baseline methods.

Main Results:

• The intelligent trainer methods demonstrated competitive autotuning capabilities.
• Achieved up to a 56% reduction in expected sampling costs without prior knowledge of optimal parameters.
• Showcased the flexibility of the RoR framework for various trainer designs and its potential for tasks with expensive hyperparameter tuning.

Conclusions:

• The RoR architecture effectively decomposes complex MBRL training into manageable RL layers.
• Automated hyperparameter optimization via the intelligent trainer significantly reduces costs and improves efficiency.
• The proposed framework offers a scalable and extensible solution for optimizing MBRL in diverse and costly applications.