Learning parametric policies and transition probability models of markov decision processes from data.

Area of Science:

• Machine Learning
• Reinforcement Learning
• Statistical Modeling

Background:

• Markov Decision Processes (MDPs) are fundamental in sequential decision-making.
• Estimating MDP models from observational data is challenging.
• Parametric assumptions on sparse features are often necessary for tractability.

Purpose of the Study:

• To develop algorithms for estimating the policy and transition probability model of an MDP from state, action, next state tuples.
• To establish theoretical guarantees on the performance of the proposed estimation methods.
• To demonstrate the practical applicability of the algorithms through real-world examples.

Main Methods:

• Regularized maximum likelihood estimation for transition probability models.
• Regularized maximum likelihood estimation for policies.
• Theoretical analysis to establish an upper bound on regret.
• Sample complexity analysis to determine the number of samples required for low regret.

Main Results:

• Two novel regularized maximum likelihood estimation algorithms are proposed.
• An upper bound on the regret of the estimated policy is derived.
• A sample complexity result demonstrates that low regret can be achieved with limited data.
• The algorithms are validated on healthcare and robot navigation problems.

Conclusions:

• The proposed methods provide an effective approach for learning MDPs from data.
• The theoretical guarantees ensure efficient learning and good performance.
• The demonstrated applications highlight the practical utility of the algorithms in complex domains.