Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimes

Area of Science:

• Artificial Intelligence in Medicine
• Critical Care Medicine
• Machine Learning for Healthcare

Background:

• Individualized ventilation strategies for critically ill patients are challenging.
• Dynamic treatment regimes (DTR) using reinforcement learning (RL) on electronic health records (EHR) show promise but are susceptible to confounding bias.
• Existing RL models may learn suboptimal policies due to confounders influencing long-term outcomes.

Purpose of the Study:

• To develop a novel deconfounding actor-critic network (DAC) for learning optimal DTR policies.
• To address confounding issues in EHR data for mechanical ventilation treatment.
• To improve patient outcomes through individualized treatment decisions.

Main Methods:

• Developed a deconfounding actor-critic network (DAC) incorporating patient resampling and confounding balance modules.
• Designed a short-term reward mechanism to capture immediate health state changes, mitigating bias from long-term outcomes.
• Introduced a policy adaptation method for transferring learned models to new, small-scale datasets.

Main Results:

• The DAC model demonstrated superior performance compared to state-of-the-art models on semi-synthetic and real-world datasets.
• The proposed methods effectively alleviated confounding issues in DTR policy learning.
• The model successfully learned optimal DTR policies for mechanical ventilation.

Conclusions:

• The DAC network offers a robust approach to learning individualized ventilation strategies.
• This method can mitigate confounding bias in EHR data, leading to more effective DTR policies.
• The developed model has the potential to significantly improve patient outcomes in critical care settings.