样本高效的深度强化学习与在线状态抽象和因果变压器模型预测.
IEEE transactions on neural networks and learning systems
|August 15, 2023
概括
深度强化学习 (RL) 样本效率通过新型抽象模型基于政策学习 (AMPL) 算法得到改善. AMPL使用状态抽象和世界模型来更快地学习,在基准任务上表现优于现有的方法.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 机器人技术 机器人技术 机器人技术
背景情况:
- 深度强化学习 (RL) 需要大量的培训数据,限制了实际应用.
- 国家抽象和世界模型为提高样本效率提供了潜力,但可能会降低性能.
研究的目的:
- 引入一种新的算法,即基于抽象模型的政策学习 (AMPL),以显著提高深度RL的样本效率.
- 解决与传统状态抽象和世界模型方法相关的性能退化问题.
主要方法:
- 开发了一种使用多步二模拟的状态抽象方法,以创建与任务相关的潜在状态空间,将马尔科夫决策过程 (MDP) 压缩成抽象的MDP.
- 设计了一种因果变压器模型预测器 (CTMP),以近似抽象的MDP,并生成长视界模拟轨迹,减少预测误差.
- 在抽象的MDP中采用了修改后的多步软演员-批评算法,具有在抽象的MDP中有效的政策学习的λ-目标.
主要成果:
- 理论分析证实AMPL在培训期间提高样本效率的能力.
- 与阿塔利游戏和DeepMind Control (DMControl) 套件上的最先进的深度RL算法相比,AMPL显示出更高的样本效率.
- 动作噪音的DMControl任务的实证结果显示AMPL对与任务无关的分心因素的稳定性.
结论:
- 拟议的AMPL算法有效地提高了深度RL中的样本效率.
- 对于复杂的RL任务,AMPL提供了强大而高性能的解决方案,即使存在观测噪声.
更多相关视频
08:05Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
Published on: June 30, 2020
7.6K
11:15Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze
Published on: February 20, 2014
13.2K
相关概念视频
Observational Learning
210
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
210
Purposive Learning
142
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
142
Associative Learning
444
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
444
State Space Representation
237
The frequency-domain technique, commonly used in analyzing and designing feedback control systems, is effective for linear, time-invariant systems. However, it falls short when dealing with nonlinear, time-varying, and multiple-input multiple-output systems. The time-domain or state-space approach addresses these limitations by utilizing state variables to construct simultaneous, first-order differential equations, known as state equations, for an nth-order system.
Consider an RLC circuit, a...
Consider an RLC circuit, a...
237
Avoidance Learning and Learned Helplessness
1.8K
Avoidance learning and learned helplessness are critical concepts in understanding behavioral responses to negative stimuli.
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
1.8K
Reinforcement Schedules
204
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
204
