CoSD: Balancing behavioral consistency and diversity in unsupervised skill discovery.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Hierarchical reinforcement learning (HRL) faces challenges with sparse rewards, making unsupervised skill discovery crucial.
• Existing skill discovery methods often prioritize diversity over internal skill consistency, leading to unstable skills.
• Overemphasis on exploration can result in skills with diffuse state visit distributions, lacking behavioral concentration.

Purpose of the Study:

• To introduce the Constrained Skill Discovery (CoSD) algorithm for balancing skill diversity and behavioral consistency.
• To address the limitations of previous methods that overemphasize skill diversity at the expense of stability.
• To improve the reliability and performance of learned skills in complex reinforcement learning environments.

Main Methods:

• CoSD integrates forward and reverse decomposition of mutual information to optimize skill learning.
• A maximum entropy policy is employed to maximize the information-theoretic objective.
• The algorithm enforces low internal state entropy for each skill, promoting behavioral consistency.

Main Results:

• Skills discovered by CoSD demonstrated more concentrated state visit distributions compared to other methods.
• CoSD achieved enhanced behavioral consistency and stability in learned skills.
• Skills exhibiting higher behavioral consistency led to superior performance in complex downstream tasks.

Conclusions:

• CoSD effectively balances skill diversity with behavioral consistency, overcoming limitations of prior unsupervised skill discovery approaches.
• The proposed method leads to more stable and reliable skills, crucial for practical applications in reinforcement learning.
• Enhanced skill consistency positively impacts performance in complex tasks, highlighting the importance of this constraint.