Quantile Markov Decision Processes

Area of Science:

• Operations Research
• Decision Theory
• Reinforcement Learning

Background:

• Traditional Markov decision processes (MDPs) focus on maximizing expected cumulative rewards.
• Many real-world scenarios require optimizing specific reward quantiles for risk-averse decision-making.
• Existing MDP frameworks may not adequately address quantile optimization objectives.

Purpose of the Study:

• To introduce and define the quantile Markov decision process (QMDP) framework.
• To develop analytical results for the optimal QMDP value function.
• To present a dynamic programming algorithm for solving QMDPs and related risk-sensitive objectives.

Main Methods:

• Development of analytical characterizations for the optimal QMDP value function.
• Design of a dynamic programming-based algorithm for policy optimization.
• Extension of the algorithm to handle Conditional Value-at-Risk (CVaR) objectives in MDPs.

Main Results:

• The paper provides theoretical insights into optimizing reward quantiles within MDPs.
• An efficient dynamic programming algorithm is proposed for finding optimal QMDP policies.
• The algorithm's applicability is demonstrated for CVaR objectives.

Conclusions:

• The QMDP framework offers a powerful approach for decision-making under quantile-based objectives.
• The presented dynamic programming algorithm effectively solves QMDPs and related risk-sensitive problems.
• The model has practical implications, as shown in an HIV treatment initiation case study.