使用强化学习实验发现决策动态的HMM
Xingche Guo1, Donglin Zeng2, Yuanjia Wang1,3
1Department of Biostatistics, Columbia University, 722 West 168th St, New York, NY, 10032, United States.
Biostatistics (Oxford, England)
|September 3, 2024
概括
大型抑郁症 (MDD) 患者表现出改变的奖励学习策略. 一个新的模型显示,MDD个体在强化学习 (RL) 中参与度低于对照组,影响决策.
科学领域:
- 计算精神病学是一种计算精神病学.
- 神经科学是一个神经科学.
- 行为经济学是一种行为经济学.
背景情况:
- 重度抑郁症 (MDD) 是导致残疾的主要原因,其异质性使诊断和治疗复杂化.
- 奖励处理中的异常越来越被认为是MDD的潜在行为标记.
- 传统的强化学习 (RL) 模型可能无法完全捕捉MDD决策的复杂性,这表明策略的切换.
研究的目的:
- 研究决策策略动态如何影响MDD患者的奖励学习.
- 提出和验证一个新的计算框架来分析MDD中的基于奖励的决策.
主要方法:
- 开发了一个新的强化学习隐藏马尔科夫模型 (RL-HMM) 框架来分析基于奖励的决策.
- RL-HMM适应了基于RL的选择和随机选择之间的战略切换,具有连续的RL状态空间和时间变化的过渡.
- 采用高效的预期最大化 (EM) 算法进行参数估计和非参数启动用于统计推理.
主要成果:
- 在广泛的模拟研究中,RL-HMM框架表现出强大的性能.
- 临床护理中抗抑郁药响应的建立调节器和生物特征的应用 (EMBARC) 研究显示,与健康对照组相比,MDD患者的RL参与率降低.
- 在MDD中较低的RL参与与在情绪冲突任务期间负面影响电路中的大脑活动有关.
结论:
- 拟议的RL-HMM框架有效地模拟了基于奖励的决策中的战略转换.
- MDD的特点是改变了奖励学习动态,特别是减少了对强化学习的参与.
- 这些发现将MDD中奖励处理异常与情感回路中的神经活动联系起来.
相关概念视频
01:20Decision Making
101
Decision-making is a fundamental cognitive process that involves evaluating alternatives and selecting among them. This process can range from simple choices, such as deciding what to wear, to complex decisions, like choosing a major in college or a career path. The complexity of the decision often dictates the approach we use, which can be broadly categorized into two types: automatic and controlled decision-making.
Automatic decision-making is fast, intuitive, and relies on gut feelings...
Automatic decision-making is fast, intuitive, and relies on gut feelings...
101
01:08Timing and Consequences on Behavior
87
In operant conditioning, the timing of reinforcement is crucial. For animals like rats and cats, immediate reinforcement (within a few seconds) is much more effective than delayed reinforcement. For example, a food reward for a rat needs to follow within 30 seconds of pressing a bar to be effective.
Humans, however, can respond to delayed reinforcers. We often make decisions between immediate small rewards and delayed larger rewards. This ability to delay gratification is a significant...
Humans, however, can respond to delayed reinforcers. We often make decisions between immediate small rewards and delayed larger rewards. This ability to delay gratification is a significant...
87
01:24Reinforcement Schedules
138
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
138
01:23Reinforcement
190
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
190
01:06Law of Effect
1.3K
B.F. Skinner, a prominent figure in behavioral psychology, introduced operant conditioning by emphasizing the role of consequences in shaping behavior. This theory builds upon the law of effect proposed by Edward Thorndike, which posits that behaviors followed by satisfying outcomes are likely to be repeated. In contrast, those followed by unsatisfying outcomes are less likely to recur.
Edward Thorndike's foundational work involved studying learning in animals, particularly using puzzle...
Edward Thorndike's foundational work involved studying learning in animals, particularly using puzzle...
1.3K
01:24Generalization, Discrimination, and Extinction
490
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
490