Unsupervised Skill Discovery Through Skill Regions Differentiation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Unsupervised reinforcement learning (RL) seeks to discover diverse behaviors to improve downstream task learning.
• Existing methods like entropy-based exploration and empowerment-driven skill learning face challenges in large state spaces and state exploration, respectively.

Purpose of the Study:

• To propose a novel skill discovery objective that enhances inter-skill state diversity and intra-skill exploration in unsupervised RL.
• To address limitations of current methods in large-scale state spaces and improve overall learning efficiency.

Main Methods:

• Developed a new skill discovery objective maximizing state density deviation between skills for diversity.
• Constructed a conditional autoencoder with soft modularization for state-density estimation in high-dimensional spaces.
• Formulated an intrinsic reward based on the autoencoder for intra-skill exploration, mimicking count-based methods in latent space.

Main Results:

• The proposed method effectively learns meaningful skills across challenging state and image-based tasks.
• Demonstrated superior performance in downstream tasks compared to existing approaches.
• Validated the effectiveness of the novel skill discovery objective and the autoencoder-based intrinsic reward.

Conclusions:

• The novel approach successfully promotes inter-skill diversity and intra-skill exploration in unsupervised RL.
• The method shows significant potential for accelerating learning in complex environments and diverse downstream applications.
• This work offers a promising direction for advancing skill discovery in reinforcement learning.