首先是多步骤:一种轻量级的深度强化学习策略,用于强大的连续控制和部分可观测性
Lingheng Meng1, Rob Gorbet2, Michael Burke3
1Electrical and Computer Engineering, University of Waterloo, 200 University Avenue West, Waterloo, N2L 3G1, ON, Canada; Electrical and Computer Systems Engineering, Monash University, 18 Alliance Lane, Clayton, 3800, VIC, Australia; Data61, CSIRO, Research Way, Calyton, 3168, VIC, Australia.
概括
接近政策优化 (PPO) 在部分可观测的环境中显示出更大的稳定性,而不是双延迟深度决定性政策梯度 (TD3) 和软行为者-批判性 (SAC). 在PPO中多步启动和适应TD3/SAC可以提高在这些具有挑战性的环境中的性能.
科学领域:
- 机器人和人工智能 机器人和人工智能
- 机器学习和控制理论
背景情况:
- 深度强化学习 (DRL) 在完全可观测的马尔科夫决策过程 (MDP) 中表现出色.
- 由于状态信息不完整,部分可观测的MDPs (POMDPs) 的性能动态发生了变化.
- 现有的DRL基准通常侧重于MDP,使POMDP的表现变得不那么被理解.
研究的目的:
- 在连续控制任务的POMDP变体上实证地比较PPO,TD3和SAC算法.
- 调查部分可观测性对领先的DRL算法的相对性能的影响.
- 确定算法适应,以提高POMDP设置中的稳定性.
主要方法:
- 对近距离政策优化 (PPO),双延迟深度决定性政策梯度 (TD3) 和软行为者-批评 (SAC) 的比较分析.
- 对标准持续控制基准的代表性POMDP配方进行评估.
- 引入多步引导到PPO和多步目标到TD3 (MTD3) 和SAC (MSAC).
主要成果:
- 在部分可观测条件下,PPO表现出卓越的稳定性和性能,与典型的MDP结果形成对比.
- 与MDP同行相比,TD3和SAC在POMDP中的表现较低.
- 修改后的TD3 (MTD3) 和SAC (MSAC) 具有多步目标,在POMDP中表现出更好的稳定性.
结论:
- 部分可观测性显著影响DLR算法性能,反转了典型的排名.
- PPO固有的多步启动在POMDP中提供了一个稳定优势.
- 将TD3和SAC等算法与多步骤目标相适应,提供了一种实用的方法,可以在部分可观测的环境中提高它们的稳定性.
相关概念视频
01:12Observational Learning
1.0K
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
1.0K
02:31Multi-Step Reactions
8.8K
Chemical reactions often occur in a stepwise fashion involving two or more distinct reactions taking place in a sequence. A balanced equation indicates the reacting species and the product species, but it reveals no details about how the reaction occurs at the molecular level. The reaction mechanism (or reaction path) provides details regarding the precise, step-by-step process by which a reaction occurs. Each of the steps in a reaction mechanism is called an elementary reaction. These...
8.8K
01:23Reinforcement
934
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
934
01:25Reinforcements in Concrete
478
Reinforced concrete is a composite material used extensively in construction, combining the compressive strength of concrete with the tensile strength of steel. This synergy is essential as concrete, while excellent at resisting compression, is weak under tension. Steel bars, or rebars, are embedded in the concrete to handle these tensile forces. The choice of steel is strategic; it shares a similar coefficient of thermal expansion with concrete, which ensures uniformity in response to...
478
01:27Corrosion of Reinforcement
586
The corrosion of steel reinforcement within concrete is a process influenced by the material's inherent properties and external factors. The high pH level of around 13, provided by calcium hydroxide present in concrete, initially protects the steel reinforcement by promoting the formation of a passive iron oxide layer on its surface.
However, over time and under certain conditions like carbonation, chloride ingress, and cracking this protective state can be compromised. Steel has areas with...
However, over time and under certain conditions like carbonation, chloride ingress, and cracking this protective state can be compromised. Steel has areas with...
586
01:24Reinforcement Schedules
513
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
513