MDP modeling for multi-stage stochastic programs

Area of Science:

• Operations Research
• Artificial Intelligence
• Control Theory

Background:

• Markov decision processes (MDPs) are foundational for sequential decision-making.
• Existing models often struggle with continuous state/action spaces and decision-dependent uncertainty.
• Handling statistical learning within stochastic programming is an ongoing challenge.

Purpose of the Study:

• To develop a flexible modeling framework for multi-stage stochastic programs with MDP features.
• To extend policy graph representations for enhanced uncertainty modeling.
• To introduce novel solution methods for these complex programs.

Main Methods:

• Incorporation of Markov decision process (MDP) elements into multi-stage stochastic programming.
• Extension of policy graphs to model decision-dependent transition probabilities and statistical learning.
• Development of novel stochastic dual dynamic programming (SDDP) variants and approximations for non-convexities.

Main Results:

• Demonstrated expressiveness of the modeling approach through examples of increasing complexity.
• Successfully extended policy graphs to handle advanced uncertainty features.
• Developed new SDDP algorithms capable of addressing non-convexities in continuous state/action spaces.

Conclusions:

• The proposed framework offers a powerful and expressive approach for complex stochastic optimization problems.
• The new solution methods provide effective tools for tackling problems previously intractable.
• This work advances the integration of MDPs and stochastic programming for enhanced decision-making.