Information-State-Based Reinforcement Learning for the Control of Partially Observed Nonlinear Systems

Area of Science:

• Control Theory
• Machine Learning
• Dynamical Systems

Background:

• Controlling nonlinear dynamical systems with partial observations presents significant challenges.
• Existing methods often struggle with model uncertainty and high dimensionality.

Purpose of the Study:

• To develop a model-based reinforcement learning (RL) approach for closed-loop control of nonlinear dynamical systems with partial nonlinear observation models.
• To transform partially observed problems into fully observed ones using an information-state approach.

Main Methods:

• An information-state approach transforms the partially observed problem into a fully observed one.
• A data-based generalization of the iterative linear quadratic regulator (ILQR) is developed for RL.
• Local linear time-varying models are approximated using autoregressive-moving-average (ARMA) models from input-output data.

Main Results:

• Equivalence between the transformed and initial partially observed optimal control problems is established.
• Conditions for solving the deterministic optimal solution are provided.
• A local perturbation feedback control law is designed for optimal solutions.

Conclusions:

• The developed RL method effectively controls complex, high-dimensional nonlinear systems.
• The approach demonstrates efficacy even with model and sensing uncertainties.
• This work offers a robust solution for partially observed optimal control problems.