Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

Area of Science:

• Cognitive Neuroscience
• Computational Neuroscience
• Motor Control

Background:

• Biological agents require adaptive behavior in dynamic environments, necessitating learning and action at multiple hierarchical levels.
• Hierarchical reinforcement learning (HRL) models this by creating temporally extended actions ('options') from sequences.
• Key questions in HRL concern option formation and neural realization.

Purpose of the Study:

• To explore how human motor sequence learning literature can inform understanding of option formation in HRL.
• To investigate the neural mechanisms underlying HRL through the lens of motor control.
• To bridge insights from motor sequence learning and reinforcement learning.

Main Methods:

• Review of existing human motor sequence literature, focusing on visuomotor tasks (e.g., discrete sequence production, M x N task).
• Analysis of how hierarchical learning and behavior are represented in sequential action tasks.
• Examination of the potential role of dorsal cortical-subcortical circuitry in supporting HRL.

Main Results:

• Motor chunks within learned sequences can be conceptualized as HRL options.
• Visuomotor sequence learning tasks provide a framework for studying hierarchical behavior.
• The dorsal cortical-subcortical pathway is implicated in supporting such hierarchical processes.

Conclusions:

• Human motor sequence learning offers valuable insights into the formation and neural basis of options in HRL.
• Integrating motor sequence literature with RL can advance experimental designs in both fields.
• This interdisciplinary approach can elucidate how complex behaviors are learned and executed hierarchically.