Hindsight-based state space exploration via counterfactual intrinsic reward assignment

Area of Science:

• Artificial Intelligence
• Machine Learning
• Causal Inference

Background:

• State space exploration is crucial but challenging in reinforcement learning.
• Current intrinsic reward functions often lack environmental knowledge, leading to inaccurate rewards.
• Hindsight knowledge from agent experience can improve intrinsic reward design.

Purpose of the Study:

• To develop a novel counterfactual intrinsic reward assignment method.
• To leverage hindsight knowledge for more effective state space exploration.
• To enhance reinforcement learning agent performance by identifying high-novelty states.

Main Methods:

• Utilizes counterfactual reasoning from causal learning to mine hindsight knowledge.
• Samples counterfactual actions and predicts next states using a structural causal model.
• Assigns intrinsic rewards based on the novelty of the actual state versus counterfactual states.

Main Results:

• Demonstrates improved state space exploration capabilities.
• Shows performance enhancements across 10 OpenAI Gym environments.
• Effectively guides agents to identify and reach novel states.

Conclusions:

• The counterfactual intrinsic reward method significantly improves reinforcement learning exploration.
• Leveraging hindsight and causal inference offers a promising direction for intrinsic reward design.
• The proposed method provides a robust framework for discovering novel states in complex environments.