Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

Area of Science:

• Robotics
• Control Theory
• Machine Learning

Background:

• Linearly Solvable Markov Decision Process (LMDP) offers a method for optimal control by transforming Bellman's equation into a linear problem.
• LMDP solutions involve eigenvalue/eigenfunction problems, requiring system dynamics and cost functions.
• Real-world robot control necessitates learning system dynamics from experience.

Purpose of the Study:

• Evaluate the LMDP framework's efficacy in practical robot control applications.
• Assess the impact of learned dynamics model accuracy on control policy derivation.
• Investigate LMDP performance with linear and bilinear dynamics models and varying cost functions.

Main Methods:

• Simulation study of a pole swing-up task to analyze learned dynamics effects.
• Real robot experiments using a Spring Dog mobile robot for a battery-catching task.
• Implementation of LMDP with linear and bilinear dynamics models and quadratic/Gaussian cost functions.

Main Results:

• A linear approximation of non-linear dynamics in simulation allowed task completion, albeit with higher costs.
• In real robot experiments, LMDP controllers with learned linear dynamics matched optimal Linear Quadratic Regulator (LQR) performance for quadratic costs.
• LMDP controllers demonstrated superior performance in non-quadratic cost tasks, even with linear dynamics models.

Conclusions:

• The LMDP framework is effective for real robot control, even when employing simplified linear models for dynamics learning.
• LMDP provides a robust approach for optimal control in robotics, adaptable to learned system dynamics.
• The study validates LMDP's practical applicability and performance across different task complexities and cost functions.