大脑中的多时间尺度强化学习
Paul Masset1,2, Pablo Tano3, HyungGoo R Kim1,2,4,5
1Department of Molecular and Cellular Biology, Harvard University, USA.
bioRxiv : the preprint server for biology
|November 28, 2023
概括
这项研究表明,强化学习代理从多个时间尺度中获益,而不仅仅是一个. 小鼠中的多巴胺神经元表现出多样化的时间折扣,表明细胞特异性质对于适应性行为至关重要.
科学领域:
- 神经科学是一个神经科学.
- 计算神经科学是一种神经科学.
- 人工智能的人工智能
背景情况:
- 适应性行为对于在复杂环境中生存至关重要.
- 强化学习 (RL) 算法模型适应性行为和多巴胺神经元活动.
- 经典RL使用单一的时间表来进行奖励折扣,这可能不反映生物复杂性.
研究的目的:
- 调查多时间尺度强化学习的计算效益.
- 探索多个时间尺度在生物强化学习中的存在和作用,特别是在多巴胺神经元中.
- 为了建模在多巴胺神经元活动中观察到的时间折扣的异质性.
主要方法:
- 在多个时间尺度上运行的模拟强化学习代理.
- 在执行行为任务的小鼠中,多巴胺神经元的电生理记录.
- 计算建模分析奖励预测错误和折扣时间常数.
主要成果:
- 具有多个时间尺度的强化学习代理显示了增强的计算效益.
- 小鼠中的多巴胺神经元在编码奖励预测错误时表现出多样化的折扣时间常数.
- 一个计算模型成功地解释了使用异质折扣因子的短暂和式多巴胺信号.
- 个体神经元折扣因子在不同任务中是一致的,表明细胞特异性质.
结论:
- 多个时间尺度是生物强化学习的基本方面,提供计算优势.
- 多巴胺神经元的功能异质性可以通过时间折扣时间尺度的变化来解释.
- 这项研究为人类和动物观察到的非指数折扣提供了机制基础.
- 这些发现为设计更有效的强化学习算法铺平了道路,这些算法受到生物系统的启发.
相关概念视频
01:35Long-term Potentiation
55.3K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
55.3K
01:24Reinforcement Schedules
149
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
149
01:01Neuroplasticity
368
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
368