Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Medicine

Background:

• Reinforcement learning (RL) offers potential for data-driven treatment policy optimization in healthcare.
• Function approximators, like deep neural networks, are crucial for complex state/action spaces but necessitate careful model selection to prevent overfitting.
• Traditional validation requires deploying policies in real environments, which is often infeasible in clinical settings.

Purpose of the Study:

• To investigate a model selection pipeline for offline RL using off-policy evaluation (OPE) as a proxy for performance validation.
• To analyze the utility and computational demands of various OPE methods for ranking candidate policies.
• To propose an efficient approach for accelerating model selection in offline RL for healthcare.

Main Methods:

• Evaluation of popular off-policy evaluation (OPE) methods for model selection in offline reinforcement learning.
• Comparative analysis of OPE methods' performance in ranking policies for a sepsis treatment task.
• Development and assessment of a novel two-stage approach to optimize the model selection process.

Main Results:

• Fitted Q Evaluation (FQE) demonstrated the highest accuracy in validation ranking among evaluated OPE methods.
• FQE incurred significant computational costs due to auxiliary model fitting and inference.
• The proposed two-stage approach effectively balances ranking accuracy with computational efficiency.

Conclusions:

• Off-policy evaluation (OPE) is a viable proxy for model selection in offline RL healthcare applications where direct validation is impractical.
• Fitted Q Evaluation (FQE) is a strong performer for policy ranking but requires optimization for computational efficiency.
• The proposed two-stage method provides a practical and accelerated solution for offline RL model selection using real-world healthcare data.