Federated Offline Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Healthcare Informatics

Background:

• Personalized medicine requires dynamic treatment regimes, often leveraging offline reinforcement learning (RL).
• Sharing sensitive healthcare data across institutions is restricted due to privacy concerns and site-specific data heterogeneity.
• Existing methods struggle to utilize distributed datasets effectively for developing robust treatment strategies.

Purpose of the Study:

• To develop a novel federated offline RL framework addressing privacy and heterogeneity in multi-site healthcare data.
• To enable the analysis of site-level features within a unified model.
• To design a communication-efficient algorithm for optimizing dynamic treatment regimes.

Main Methods:

• Proposed a multi-site Markov decision process model accommodating both homogeneous and heterogeneous site effects.
• Developed the first federated policy optimization algorithm for offline RL with guaranteed sample complexity.
• Algorithm requires only a single round of communication via summary statistics exchange.

Main Results:

• The proposed federated offline RL algorithm demonstrates theoretical guarantees on policy suboptimality, comparable to centralized data scenarios.
• Extensive simulations confirm the algorithm's effectiveness in learning optimal policies.
• The method was successfully applied to a multi-site sepsis dataset.

Conclusions:

• Federated offline RL is a viable approach for personalized medicine with distributed, private healthcare data.
• The proposed algorithm offers an efficient and effective solution for multi-site treatment regime optimization.
• This work facilitates the clinical application of advanced RL techniques in real-world healthcare settings.