Value-free reinforcement learning: policy optimization as a minimal model of operant behavior

Area of Science:

• Cognitive Neuroscience
• Neuroeconomics
• Computational Psychiatry

Background:

• Reinforcement learning (RL) is crucial for modeling learning and decision-making.
• Current research predominantly uses value-based RL models, assuming decisions are based on comparing action values.
• An alternative, policy-gradient RL, optimizes behavior directly without explicit value-learning.

Purpose of the Study:

• To review behavioral and neural evidence.
• To demonstrate that policy-gradient models offer a more parsimonious explanation for observed findings compared to value-based models.
• To highlight the utility of policy-gradient models in understanding operant behavior.

Main Methods:

• Literature review of recent behavioral and neural findings.
• Comparative analysis of explanatory power between value-based and policy-gradient RL models.
• Synthesis of evidence supporting policy-gradient approaches.

Main Results:

• Several behavioral and neural findings are more simply explained by policy-gradient models.
• Policy-gradient models bypass the need for intermediate value-learning steps.
• Direct policy optimization provides a compelling account of operant behavior.

Conclusions:

• Policy-gradient reinforcement learning presents a lightweight and effective alternative to value-based models.
• These models offer a valuable framework for understanding decision-making processes.
• The study advocates for broader consideration of policy-gradient models in cognitive neuroscience and neuroeconomics.