大脑中的多时间尺度强化学习
Paul Masset1,2,3,4, Pablo Tano5, HyungGoo R Kim6,7,8,9
1Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA. paul.masset@mcgill.ca.
Nature
|June 4, 2025
概括
动物和人类使用多个时间表来进行强化学习, 而不是一个时间表. 这项研究揭示了小鼠中的多巴胺神经元表现出多样化的时间折扣,改善了适应性行为,并告知了新的学习算法.
科学领域:
- 神经科学
- 计算神经科学
- 机器学习
背景情况:
- 在复杂的环境中适应行为需要最大限度的回报.
- 强化学习 (RL) 模拟了这种适应性行为,并描述了多巴胺基神经元活动.
- 传统的RL使用单一的折扣因子来获得未来的奖励.
研究的目的:
- 研究多个时间尺度在生物强化学习中的作用.
- 在各种时间尺度上探索代理学习的计算效益.
- 描述多巴胺能神经元的时间折扣特性.
主要方法:
- 开发了多个学习时间表的强化学习剂.
- 在两项行为任务中记录了小鼠的多巴氨基神经元活动.
- 模拟奖励预测错误和神经反应中的时间折扣.
主要成果:
- 具有多个时间尺度的强化剂显示了增强的计算效益.
- 小鼠中的多巴胺神经元显示出各种折扣时间常数.
- 一个模型解释了时间折扣中的神经异质性,包括多巴胺.
- 各个神经元折扣因子在各个任务中一致,表明细胞特异性.
结论:
- 多个时间尺度对于有效的生物强化学习至关重要.
- 时间折扣中的多巴胺基神经元异质性为非指数级奖励估值提供了机制基础.
- 这些发现为理解多巴胺功能和设计先进的RL算法提供了新的框架.
相关概念视频
Long-term Potentiation
55.9K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
55.9K
Reinforcement Schedules
243
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
243
Neuroplasticity
808
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
808


