Hindsight-based state space exploration via counterfactual intrinsic reward assignment

State space exploration has been a core challenge in reinforcement learning, due to the difficulty in designing an intrinsic reward function that can guide agents to precisely find high-novelty states in the state space. Most existing methods design the reward function without fully utilizing any knowledge concerning environments, which often produces inaccurate reward signals. Instead, this paper assumes the hindsight knowledge extracted from the agent's previous exploration experience can help intrinsic reward designs. Therefore, the counterfactual intrinsic reward assignment method is presented, which uses the counterfactual reasoning mechanism from the causal learning field to mine hindsight knowledge from previously explored states, and utilizes this knowledge for better intrinsic reward assignment to the agent. The core idea is to "backtrack" the agent's obtained exploration result and "reflect on" whether a more novel state could have been discovered, if the agent had selected another action at the time. Concretely, the method first samples a batch of "counterfactual" actions that differ from the current action from a policy, then uses a structural causal model to predict their corresponding next states. Subsequently, the method will give a large reward if the state actually explored is more novel than that in counterfactual reasoning, otherwise a low reward, thereby "rewarding" agents to effectively identify and find novel states in the state space. Simulations show the effectiveness in improving state space exploration, with the performance enhanced over 10 OpenAI Gym environments.