Semi-Supervised Off-Policy Reinforcement Learning and Value Estimation for Dynamic Treatment Regimes

Area of Science:

• * Computational statistics and machine learning applied to healthcare.
• * Development of advanced algorithms for personalized medicine.

Background:

• * Reinforcement learning (RL) shows potential for dynamic treatment regimes, but relies on accurate health outcome data.
• * Clinical notes often contain outcome information, but manual extraction is resource-intensive, leading to small labeled datasets.
• * Existing methods struggle with limited labeled data for training effective RL models.

Purpose of the Study:

• * To develop a semi-supervised learning (SSL) approach for reinforcement learning (RL) in dynamic treatment regimes.
• * To leverage both small labeled datasets with observed outcomes and large unlabeled datasets with outcome surrogates.
• * To address challenges in generalizing SSL to dynamic treatment regimes, including unknown feature distributions and non-informative surrogate variables.

Main Methods:

• * Proposed a semi-supervised, efficient approach to Q-learning and doubly robust off-policy value estimation.
• * Developed a modified SSL framework to handle outcome surrogates predictive but not policy-informative.
• * Provided theoretical analysis of Q-function and value-function estimators to quantify SSL efficiency gains.

Main Results:

• * The proposed SSL method demonstrates at least equivalent efficiency compared to purely supervised approaches.
• * The method is robust to potential bias arising from mis-specified imputation models.
• * Theoretical results quantify the efficiency gains achieved by incorporating unlabeled data via SSL.

Conclusions:

• * SSL offers an effective strategy to enhance RL for dynamic treatment regimes using limited labeled clinical data.
• * The proposed methods provide efficient and robust value estimation in challenging healthcare data scenarios.
• * This work advances the application of machine learning in personalized medicine by improving data utilization.