Invariant Policy Learning: A Causal Perspective.

Area of Science:

• Machine Learning
• Causal Inference
• Reinforcement Learning

Background:

• Contextual bandit and reinforcement learning algorithms are effective in online advertising and recommender systems.
• Widespread adoption in high-stakes domains like healthcare is limited due to the assumption of static underlying mechanisms.
• Environmental shifts across different settings can invalidate existing static environment assumptions.

Purpose of the Study:

• To address the challenge of environmental shifts within the framework of offline contextual bandits.
• To propose a novel approach for handling dynamic changes in underlying mechanisms across environments.
• To enhance the generalizability of contextual bandit policies in real-world, non-static systems.

Main Methods:

• Framing the environmental shift problem through the lens of causality.
• Introducing multi-environment contextual bandits to accommodate changes in underlying mechanisms.
• Adopting the concept of invariance from causality literature and defining policy invariance.

Main Results:

• Policy invariance is particularly relevant when unobserved variables are present.
• An optimal invariant policy is demonstrated to generalize across environments under specific assumptions.
• The proposed framework allows for changes in underlying mechanisms, overcoming limitations of static models.

Conclusions:

• The developed multi-environment contextual bandit approach offers a robust solution for systems with environmental shifts.
• The concept of policy invariance provides a theoretical guarantee for generalization in the presence of unobserved confounders.
• This work paves the way for wider adoption of contextual bandits in high-stakes applications like healthcare by addressing mechanism shifts.