Harnessing Causality in Reinforcement Learning With Bagged Decision Times.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Science

Background:

• Reinforcement learning (RL) traditionally assumes Markovian decision processes.
• Real-world problems, such as mobile health interventions, often exhibit non-Markovian and non-stationary dynamics within decision periods (bags).
• Existing RL methods struggle with these complex temporal dependencies.

Purpose of the Study:

• To develop an online reinforcement learning (RL) algorithm capable of maximizing rewards in scenarios with bagged decision times and non-Markovian transitions.
• To address the challenge of jointly optimizing actions within a bag that collectively influence a single reward.
• To adapt RL for periodic Markov decision processes (MDPs) with intra-period non-stationarity.

Main Methods:

• Utilized expert-provided causal directed acyclic graphs (DAGs) to model dependencies within bags.
• Constructed states as dynamical Bayesian sufficient statistics of historical data to ensure Markovian transitions.
• Formulated the problem as a periodic MDP and generalized Bellman equations for online RL optimization.
• Evaluated the proposed method on mobile health trial data.

Main Results:

• The proposed state construction method ensures Markovian state transitions within and across bags.
• The developed online RL algorithm effectively handles non-stationarity within periodic MDPs.
• The constructed state was shown to achieve the maximal optimal value function for the periodic MDP.
• Successful evaluation on mobile health testbeds demonstrated practical applicability.

Conclusions:

• The novel RL framework successfully addresses non-Markovian and non-stationary dynamics in bagged decision time problems.
• The DAG-based state construction provides an effective way to manage complex temporal dependencies.
• The method offers a promising approach for optimizing sequential decision-making in domains like mobile health.