First return, then explore.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) aims for autonomous decision-making but struggles with sparse or deceptive reward signals.
• Effective environment exploration is crucial for RL but remains a significant challenge.
• Existing RL algorithms often forget previously visited states or fail to revisit them before exploring new ones.

Purpose of the Study:

• To address the challenges of detachment and derailment in reinforcement learning exploration.
• To introduce a novel algorithm, Go-Explore, designed for more effective environment exploration.
• To demonstrate the efficacy of Go-Explore on complex games and robotics tasks.

Main Methods:

• Introduced Go-Explore, an algorithm family emphasizing remembering promising states and returning to them before exploration.
• Applied Go-Explore to previously unsolved Atari games and hard-exploration benchmarks.
• Tested Go-Explore on a sparse-reward pick-and-place robotics task.
• Integrated goal-conditioned policies to further enhance exploration efficiency and handle stochasticity.

Main Results:

• Go-Explore successfully solved all previously unsolved Atari games.
• Achieved state-of-the-art performance on hard-exploration games, with significant improvements on Montezuma's Revenge and Pitfall.
• Demonstrated practical applicability on a challenging sparse-reward robotics task.
• Goal-conditioned policies enhanced Go-Explore's exploration efficiency and robustness to stochasticity.

Conclusions:

• The principles of remembering, returning to, and exploring from states are a powerful and general approach to RL exploration.
• Go-Explore offers substantial performance gains, suggesting a critical pathway toward more intelligent learning agents.
• The findings highlight the importance of structured exploration strategies in overcoming RL limitations.