DDRL:Dyna-Based Discriminative Reinforcement Learning for Optimizing Sepsis Treatment Pathways in Offline...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Reinforcement Learning for Healthcare

Background:

• Automated sepsis treatment policies using Reinforcement Learning (RL) aim to enhance care quality and mitigate physician shortages.
• Offline RL faces challenges with limited exploration, leading to Q-value overestimation and suboptimal policies deviating from physician practices.

Purpose of the Study:

• To develop a Dyna-Based Discriminative Reinforcement Learning (DDRL) method for optimal sepsis treatment policy learning.
• To ensure the learned policy aligns with established physician treatment strategies.
• To overcome limitations of offline RL in healthcare settings.

Main Methods:

• DDRL integrates Electronic Medical Record (EMR) data with simulated treatment episodes.
• A Discriminator component is employed to suppress Q-values for out-of-distribution treatments.
• This approach mitigates Q-value overestimation and reduces policy deviation from physician behavior.

Main Results:

• DDRL demonstrated superior performance compared to Conservative Q-Learning (CQL) and physician policies.
• Expected returns for DDRL were 7.29 (Asan Medical Center) and 4.55 (Ajou University Hospital).
• Cosine similarity between DDRL and physician policies reached 81.68% and 90.90%, significantly outperforming CQL.

Conclusions:

• DDRL effectively learns optimal sepsis treatment policies that align with physician expertise.
• The method successfully addresses offline RL limitations by incorporating EMR data and simulation.
• DDRL shows promise for improving automated clinical decision-making in sepsis management.