Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for...

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Machine Learning

Background:

• Human intelligence excels at compositional generalization, recombining known elements for novel problems, a challenge for current reinforcement learning (RL) agents.
• RL agents struggle to independently transfer reward and transition functions, limiting their ability to generalize across diverse tasks like navigation with various transport modes.
• Existing model-based RL research has not fully addressed how a single agent can learn and transfer both individual task components and entire learned structures.

Purpose of the Study:

• To develop a hierarchical reinforcement learning (RL) agent that can learn and transfer both individual task components (reward and transition functions) and entire task structures.
• To enable agents to achieve compositional generalization by inferring task components and their covariances from data.
• To address the limitations of current RL approaches in simultaneously learning and transferring independent and interdependent task elements.

Main Methods:

• Proposed a hierarchical RL agent employing a non-parametric Bayesian model for task inference.
• Utilized a hierarchical Dirichlet process to maintain a factorized representation of task components.
• Incorporated a standard Dirichlet process to model covariances between task components, enabling joint transfer of structures.

Main Results:

• The agent demonstrated improved generalization and transfer capabilities on various navigation tasks with diverse statistical correlations between task components.
• The approach successfully learned and transferred both individual task components and entire task structures.
• Validation on complex, hierarchical tasks with goal/subgoal structures showed enhanced performance.

Conclusions:

• The proposed hierarchical RL agent effectively addresses the challenge of compositional generalization in artificial intelligence.
• The model's ability to infer and transfer both individual components and combined structures offers a significant advancement in RL.
• The findings suggest potential biological plausibility, with implications for understanding cortico-striatal gating circuits in the brain.