Diminishing Return of Value Expansion Methods

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-based reinforcement learning (MBRL) uses dynamics models to improve sample efficiency.
• Model accuracy and compounding errors are perceived as key limitations in MBRL.

Purpose of the Study:

• Empirically investigate sample efficiency gains from improved dynamics models in model-based value expansion.
• Identify limitations in model-based value expansion methods beyond model accuracy.

Main Methods:

• Utilized oracle dynamics models to eliminate compounding errors in model-based value expansion.
• Compared performance of model-based methods with varying rollout horizons and model accuracies against model-free counterparts.

Main Results:

• Longer rollout horizons enhance sample efficiency, but with diminishing returns.
• Increased model accuracy provided only marginal sample efficiency gains over learned models.
• Model-free value expansion methods achieved comparable performance without computational overhead.

Conclusions:

• Model accuracy is not the sole limiting factor for sample efficiency in model-based value expansion.
• Even perfect models do not guarantee superior sample efficiency, indicating other performance constraints.
• Future research should focus on identifying these alternative limiting factors in MBRL.