有效的线下到线上强化学习,预先降低了分布之外的Q值
Tao Wang1, Xiangfeng Luo1, Zhenyu Zhang1
1School of Computer Engineering and Science, Shanghai University, 99 Shangda Road BaoShan District, Shanghai, 200444, China.
概括
预先减少的Q学习 (PRQL) 通过稳定在线微调来增强线下强化学习 (RL). 这种方法减少了对分销之外的行动的过高估计,有效地提高了政策绩效.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 机器人技术 机器人技术 机器人技术
背景情况:
- 线下强化学习 (RL) 在很大程度上依赖于培训数据的质量.
- 在专家数据稀缺的情况下,在线微调对于改善政策至关重要.
- 直接扩展离线算法或使用传统的政策之外的微调方法可以导致缓慢的改进或最初的性能下降,这是由于过度估计的分发外行动造成的.
研究的目的:
- 通过线下强化学习来学习的微调政策,开发一种高效稳定的方法.
- 在政策微调中,应对缓慢改进和初始绩效下降的挑战.
- 提出预先缩小的Q学习 (PRQL) 作为一种用于增强在线微调的新方法.
主要方法:
- PRQL训练了一项初步策略,以识别离线Q网络过度估计的行为.
- 一个中间的Q网络被训练来降低这些过度估计的行动的估计值.
- 然后,该方法使用新的在线数据进行微调,并将内存规范化术语纳入内存,以保存已学到的值.
主要成果:
- 对连续控制任务的实验表明,PRQL的表现优于现有的基线.
- 在微调过程中,PRQL 实现了显著的性能改进.
- 与其他方法相比,拟议的方法显示出相对较小的初始政策下降.
结论:
- PRQL提供了一种有效的解决方案,用于在线强化学习中稳定有效地改进政策.
- 该方法成功地减轻了超估分销行动的负面影响.
- 对于需要在线适应的强化学习的实际应用来说,PRQL是一个有前途的进步.
相关概念视频
01:00Detection of Gross Error: The Q Test
6.4K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.4K
02:35Reaction Quotient
49.1K
The status of a reversible reaction is conveniently assessed by evaluating its reaction quotient (Q). For a reversible reaction described by m A + n B ⇌ x C + y D, the reaction quotient is derived directly from the stoichiometry of the balanced equation as
49.1K
01:24Reinforcement Schedules
242
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
242
01:09Decision Making: P-value Method
5.7K
The process of hypothesis testing based on the P-value method includes calculating the P- value using the sample data and interpreting it.
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
5.7K
01:13Randomized Experiments
7.2K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
7.2K
01:23Reinforcement
341
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
341