CausalCOMRL: Context-based offline meta-reinforcement learning with causal representation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline meta-reinforcement learning (OMRL) uses offline datasets for task representation learning.
• Existing methods suffer from spurious correlations due to confounders, limiting generalizability.
• Confounder-induced correlations degrade policy performance when test tasks differ from training tasks.

Purpose of the Study:

• To propose CausalCOMRL, a novel context-based OMRL method integrating causal representation learning.
• To address spurious correlations and enhance the generalizability of reinforcement learning agents.
• To improve the distinction of task representations across different tasks.

Main Methods:

• Causal representation learning to uncover causal relationships among task components.
• Mutual information optimization and contrastive learning to enhance task representation distinctiveness.
• Soft Actor-Critic (SAC) algorithm for policy optimization using causal task representations.

Main Results:

• CausalCOMRL demonstrates superior performance compared to existing methods across most meta-RL benchmarks.
• The method effectively mitigates the negative impact of spurious correlations.
• Causal task representations lead to improved generalizability in reinforcement learning agents.

Conclusions:

• CausalCOMRL offers a robust approach to context-based OMRL by leveraging causal inference.
• The integration of causal representation learning significantly enhances agent performance and generalizability.
• This work advances the field of OMRL by providing a method to overcome confounder-induced limitations.