REINFORCEMENT LEARNING FOR INDIVIDUAL OPTIMAL POLICY FROM HETEROGENEOUS DATA.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline reinforcement learning (RL) seeks optimal policies from pre-collected data.
• Learning from heterogeneous data is a key challenge in offline RL.
• Existing methods often yield suboptimal policies for heterogeneous populations.

Purpose of the Study:

• To propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs).
• To address the limitations of traditional methods in handling diverse population data.
• To develop an algorithm that efficiently estimates individual Q-functions.

Main Methods:

• Developed a heterogeneous model with individual latent variables.
• Introduced the Penalized Pessimistic Personalized Policy Learning (P4L) algorithm.
• Guaranteed a fast rate on average regret under partial coverage assumptions.

Main Results:

• The proposed framework efficiently estimates individual Q-functions.
• The P4L algorithm demonstrates superior numerical performance in simulations and real-world applications.
• Achieved better policy optimization for heterogeneous populations compared to existing methods.

Conclusions:

• The individualized offline policy optimization framework effectively handles heterogeneous data in MDPs.
• The P4L algorithm offers a robust solution for personalized policy learning.
• The study highlights the importance of individualized approaches in offline RL for diverse populations.