Prospective and retrospective temporal difference learning.

Area of Science:

• Behavioral neuroscience
• Computational neuroscience
• Reinforcement learning

Background:

• Monkeys exhibit maladaptive behavior in tasks with delayed rewards, potentially due to Pavlovian influences.
• Previous research identified that task state performance depends on recent history, contradicting standard reinforcement learning (RL) models.
• An alternative temporal difference (TD) model was proposed to account for this history-dependent behavior.

Purpose of the Study:

• To demonstrate that a standard TD model can replicate history-dependent behavior observed in monkeys.
• To show that this standard model avoids negative consequences for choice prediction.
• To highlight the role of average reward per step as a crucial baseline in RL.

Main Methods:

• Analysis of behavioral data from tasks with systematically delayed rewards.
• Modeling using a standard temporal difference (TD) reinforcement learning framework.
• Investigating the influence of average reward per step as a dynamic baseline.

Main Results:

• A standard TD model successfully replicates the observed history-dependent behavior in monkeys.
• The model demonstrates that average reward per step acts as a critical baseline, influencing predictions.
• Subtle historical changes to this baseline significantly impact behavioral predictions.

Conclusions:

• Standard TD models, incorporating average reward per step, can explain complex behavioral phenomena.
• This approach reconciles observed monkey behavior with established RL theory.
• Understanding the baseline's dynamic nature is key to accurate prediction of behavior in delayed reward tasks.