A Survey on Causal Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Causal Inference

Background:

• Reinforcement learning (RL) excels in sequential decision-making but struggles with data inefficiency and interpretability.
• Causality literature offers insights to enhance RL's capabilities.
• Causal reinforcement learning (CRL) emerges as a promising interdisciplinary field.

Purpose of the Study:

• To systematically review and categorize existing causal reinforcement learning (CRL) methods.
• To analyze various formalizations of CRL models, including MDP, POMDP, MAB, IL, and DTR.
• To investigate the potential of causal inference to address key RL challenges.

Main Methods:

• Categorization of CRL approaches based on the availability of causality-based information (a priori vs. not).
• Analysis of different model formalizations: Markov decision process (MDP), partially observed MDP (POMDP), multiarmed bandits (MABs), imitation learning (IL), and dynamic treatment regime (DTR).
• Review of evaluation matrices, open-source resources, and emerging applications.

Main Results:

• CRL methods are classified into two main categories based on causal information.
• Diverse causal graphical illustrations are presented for various RL models.
• A comprehensive overview of the current landscape of CRL is provided.

Conclusions:

• Causality offers significant potential to improve RL's data efficiency and interpretability.
• Further research in CRL is crucial for advancing sequential decision-making.
• Emerging applications and future research directions in CRL are identified.