Where Does Value Come From?

Area of Science:

• Computational neuroscience
• Behavioral economics
• Cognitive psychology

Background:

• Reinforcement learning (RL) is a powerful framework for understanding biological brains and artificial agents.
• However, current RL models face limitations in explaining animal behavior in natural, complex environments.

Purpose of the Study:

• To highlight open challenges for RL as a model of animal behavior in natural environments.
• To explore how biological systems design external reward functions and account for context-sensitive valuation.
• To propose a new framework for understanding animal decision-making.

Main Methods:

• Review and synthesis of existing theories in reinforcement learning, psychology, behavioral ecology, and economics.
• Integration of old and new theoretical proposals regarding internal states and goal-directed behavior.
• Examination of recent brain-imaging findings related to value-guided decision-making.

Main Results:

• Biological systems may not rely solely on external reward functions; internal state regulation is crucial.
• Animals likely minimize the 'distance' between current and desired internal states across multiple value dimensions.
• This internal state-based framework offers a unified explanation for diverse behavioral phenomena.

Conclusions:

• A novel RL framework, focusing on internal state regulation and goal-seeking across value dimensions, provides a more comprehensive model of animal behavior.
• This approach reconciles findings from psychology, behavioral ecology, economics, and neuroimaging.
• Future research should further explore the computational and neural mechanisms underlying internal state-based decision-making in animals.