改善在线强化学习的政策利用,即时回顾行动
Gong Gao1, Weidong Zhao1, Xianhui Liu1
1School of Computer Science, Tongji University, China.
概括
即时回顾行动 (IRA) 通过加强探索和政策更新来改善在线强化学习 (RL). 这种新的算法加速了学习效率和连续控制任务的最终性能.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 机器人技术 机器人技术 机器人技术
背景情况:
- 在线强化学习 (RL) 算法面临着由于探索效率低下和更新延迟而导致政策利用速度缓慢的挑战.
- 现有的方法在动态环境中难以实现快速适应和有效学习.
研究的目的:
- 引入一个新的算法,即时回顾行动 (IRA),旨在克服当前基于价值的在线RL方法的局限性.
- 提高学习效率和持续控制任务的最终性能.
主要方法:
- 为歧视性的Q网络表示提出Q表示差异演变 (RDE).
- 实施贪行动指导 (GAG) 通过政策约束的行动回溯.
- 引入即时政策更新 (IPU) 机制以增加更新频率.
- 利用k-最近邻国行动价值估计来准确的政策学习.
主要成果:
- 在八个MuJoCo连续控制任务中,IRA显著提高了学习效率和最终绩效.
- 该算法显示了政策利用的增强和更快的适应.
- 在IRA中,早期阶段的培训保守主义有助于减轻基于价值的RL中的高估偏见.
结论:
- 在线强化学习方面,IRA提供了实质性的进步,特别是在持续控制问题上.
- 提出的方法 (RDE,GAG,IPU) 共同有助于更有效和高效的RL代理人培训.
- 对于开发更强大,更高性能的RL系统来说,IRA是一个有前途的方向.
相关概念视频
Fixed Action Patterns
17.7K
A fixed action pattern (FAP) is a specific, hard-wired sequence of behaviors that occurs in response to an external stimulus, called a sign stimulus. The behavior is “fixed” because it is essentially unchangeable—proceeding similarly across individuals of a species every time it occurs.
17.7K
Reinforcement
952
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
952
Corrosion of Reinforcement
590
The corrosion of steel reinforcement within concrete is a process influenced by the material's inherent properties and external factors. The high pH level of around 13, provided by calcium hydroxide present in concrete, initially protects the steel reinforcement by promoting the formation of a passive iron oxide layer on its surface.
However, over time and under certain conditions like carbonation, chloride ingress, and cracking this protective state can be compromised. Steel has areas with...
However, over time and under certain conditions like carbonation, chloride ingress, and cracking this protective state can be compromised. Steel has areas with...
590
Reinforcement Schedules
527
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
527
Reinforcements in Concrete
484
Reinforced concrete is a composite material used extensively in construction, combining the compressive strength of concrete with the tensile strength of steel. This synergy is essential as concrete, while excellent at resisting compression, is weak under tension. Steel bars, or rebars, are embedded in the concrete to handle these tensile forces. The choice of steel is strategic; it shares a similar coefficient of thermal expansion with concrete, which ensures uniformity in response to...
484
Fiber Reinforced Concrete
423
Fiber-reinforced concrete significantly enhances the structural and nonstructural properties of traditional concrete by incorporating fibers like steel, glass, and polymers. These fibers, varying from natural ones such as sisal and cellulose to manufactured ones like polypropylene and Kevlar, are mixed into hydraulic cement with aggregates. Steel fibers, often preferred for their robustness, contribute to improved ductility, toughness, and post-cracking performance. The concrete is classified...
423


