Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Behavior is often explained by competing brain systems, with model-free (MF) and model-based (MB) reinforcement learning agents hypothesized to represent automaticity and deliberation.
• Static mixture-of-agents (MoA) models capture these differences but cannot account for dynamic strategy shifts.

Purpose of the Study:

• To develop a novel computational framework, the mixture-of-agents hidden Markov model (MoA-HMM), capable of capturing dynamic strategy shifts in behavior.
• To investigate the temporal dynamics of agent contributions and their relationship to behavioral and neural changes during a reward-guided task.

Main Methods:

• Developed the MoA-HMM to simultaneously infer action values from multiple agents and model the temporal dynamics of underlying hidden states.
• Applied the MoA-HMM to behavioral data from rats performing a multi-step, reward-guided task.

Main Results:

• The MoA-HMM revealed a progression of within-session strategies in rats, shifting from initial MB exploration to MB exploitation, and finally to reduced engagement.
• Inferred hidden states predicted significant changes in response time and orbitofrontal cortex (OFC) neural encoding during the task.

Conclusions:

• The MoA-HMM provides a dynamic framework for understanding how competing learning systems interact and shift over time.
• The identified hidden states represent genuine shifts in cognitive dynamics, influencing both behavior and neural representations.