Continuous action deep reinforcement learning for propofol dosing during general anesthesia

Area of Science:

• Anesthesiology and Artificial Intelligence
• Machine Learning in Medicine
• Pharmacodynamics and Pharmacokinetics

Background:

• Anesthesiologists manage multiple patient care aspects during general anesthesia.
• Automating hypnotic agent administration can enhance control over unconsciousness levels.
• Reinforcement learning (RL) offers a promising approach for automated anesthetic drug delivery systems.

Purpose of the Study:

• To develop and evaluate a continuous-action deep reinforcement learning (RL) agent for automated anesthetic dosing.
• To compare the performance of the RL agent against a traditional proportional-integral-derivative (PID) controller.
• To assess the clinical viability and interpretability of the proposed automated system.

Main Methods:

• Utilized an actor-critic RL paradigm with a policy network and a value network.
• Trained and tested three RL agent versions with varied reward functions on simulated data.
• Employed Shapley additive explanations for understanding agent decision-making and validated on retrospective clinical cases.

Main Results:

• The deep RL agent significantly outperformed the PID controller in performance error.
• The RL agent rewarded for minimizing total anesthetic doses demonstrated superior performance across simulations.
• Agent-recommended doses were consistent with anesthesiologist administration in real-world clinical data.

Conclusions:

• This study presents the first fully continuous deep RL algorithm for automated anesthetic dosing.
• Flexible reward function design allows for optimization of anesthetic practices and performance.
• The agent's dosing decisions align with established best practices in anesthesia care, confirmed through interpretability analysis and clinical data validation.