Finding structure in multi-armed bandits.

Area of Science:

• Cognitive Science
• Neuroscience
• Reinforcement Learning

Background:

• Human reward search is often modeled using multi-armed bandit tasks.
• Standard models assume independent reward distributions for each option, which is unrealistic.
• Real-world options often share underlying structures that influence learning.

Purpose of the Study:

• To investigate how generalization guides exploration in structured environments.
• To understand if humans utilize latent functional structure in reward-seeking tasks.
• To examine learning-to-learn effects across repeated exploration rounds.

Main Methods:

• Utilized structured multi-armed bandit tasks with options correlated by a latent function.
• Focused on bandits where rewards were linear functions of spatial position.
• Conducted 5 experiments to observe human exploration strategies and compared computational models.

Main Results:

• Participants demonstrated the use of functional structure to guide exploration.
• Evidence of a learning-to-learn effect was observed, with faster function identification over time.
• Findings held true for both linear and non-linear reward functions, ruling out heuristic explanations.

Conclusions:

• Human reinforcement learning effectively leverages latent structure for efficient exploration.
• Optimal models of human behavior integrate function learning, reward distribution clustering, and uncertainty-guided exploration.
• This suggests sophisticated cognitive mechanisms underlie human decision-making in complex reward landscapes.