Risk-sensitive reinforcement learning.

Area of Science:

• Neuroscience
• Cognitive Science
• Machine Learning

Background:

• Sequential decision-making in uncertain environments is a complex cognitive task.
• Traditional reinforcement learning models often assume risk neutrality, failing to capture human behavior.
• Prospect theory describes systematic deviations from rational choice, including risk preferences and probability weighting.

Discussion:

• Applying utility functions to temporal difference errors in reinforcement learning allows for risk-sensitive agents.
• This approach models human risk preferences for gains and losses, aligning with prospect theory.
• A novel risk-sensitive Q-learning algorithm is derived and proven to converge, even with unknown transition probabilities.

Key Insights:

• The risk-sensitive reinforcement learning framework significantly improves the fit to human behavioral data in sequential investment tasks.
• It provides an interpretation of subject responses consistent with prospect theory's principles.
• Risk-sensitive temporal difference errors correlate with brain activity in the ventral striatum, suggesting a neural basis for risk-sensitive learning.

Outlook:

• This framework offers a powerful tool for quantifying and understanding human decision-making under risk.
• Future research can explore applications in various domains, including economics, psychology, and artificial intelligence.
• Investigating the neural underpinnings of risk-sensitive learning can lead to more sophisticated computational models of the brain.