Human Strategy Adaptation in Reinforcement Learning Resembles Policy Gradient Ascent

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Neuroscience

Background:

• Adapting learning strategies is crucial for intelligence but lacks quantitative frameworks in biological agents.
• Existing computational models often assume fixed strategies or use task-optimized networks, failing to explain strategy refinement through experience.

Purpose of the Study:

• To develop a quantitative framework for characterizing how biological agents adapt their learning strategies.
• To investigate if human strategy adaptation resembles gradient-based optimization principles.

Main Methods:

• Introduced DynamicRL, a neural network framework to track evolving learning parameters (learning rates, decision temperatures) in participants.
• Evaluated DynamicRL across four diverse bandit tasks.

Main Results:

• DynamicRL outperformed traditional reinforcement learning models with fixed parameters.
• Human learning strategy adaptation showed trajectories that systematically increased expected rewards.
• Strategy parameter updates aligned with policy gradient ascent directions and operated across multiple timescales.

Conclusions:

• Humans dynamically adapt their reinforcement learning strategies, aligning with gradient-based optimization principles.
• The DynamicRL framework provides a generalizable method for studying meta-learning trajectories in biological agents.
• This research bridges theories of biological and artificial intelligence by quantifying adaptive behavior optimization through experience.