Stochastic Optimal Control for Multivariable Dynamical Systems Using Expectation Maximization

Area of Science:

• Robotics and Control Systems
• Machine Learning
• Operations Research

Background:

• Trajectory optimization is a core problem in stochastic optimal control (SOC).
• Analytical solutions for these problems are often intractable due to their nature as partially observable Markov decision processes (MDPs).
• Existing methods often rely on approximations, driving the need for more effective techniques.

Purpose of the Study:

• To develop an effective trajectory optimization approach for dynamical systems with measurement noise.
• To reformulate stochastic control problems within a reinforcement learning framework.
• To introduce and validate the SOC-EM iterative optimization paradigm.

Main Methods:

• Reformulation of stochastic control problems in a reinforcement learning setting.
• Development of an iterative trajectory optimization algorithm named SOC-expectation maximization (SOC-EM).
• Theoretical analysis of control parameter estimate uniqueness and control covariance matrix.

Main Results:

• The proposed SOC-EM method demonstrates superior performance in reducing cumulative cost-to-go.
• Empirical and theoretical validation confirms the effectiveness of the SOC-EM approach.
• Novel theoretical contributions regarding the uniqueness of control parameter estimates and stochasticity handling.

Conclusions:

• The SOC-EM paradigm offers a powerful and effective solution for trajectory optimization in noisy dynamical systems.
• This approach successfully integrates benefits from both conventional optimal control and maximum likelihood methods.
• The work provides significant theoretical insights into the control of stochastic systems.