偏差减少域随机化用于强化学习与政策梯度的强化学习
IEEE transactions on pattern analysis and machine intelligence
|November 6, 2023
概括
域随机化 (DR) 增强了深度强化学习的概括性,但增加了训练差异. 本研究引入了一种减少差异域随机化 (VRDR) 方法,具有最佳基线,以提高机器人控制任务中的样本效率和政策性能.
科学领域:
- 机器人技术 机器人技术 机器人技术
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 域随机化 (DR) 对于通过引入环境多样性来提高深度强化学习 (DRL) 政策的泛化能力至关重要.
- 然而,由于环境变化的增加,DR加剧了政策梯度变化,阻碍了DRL培训中的样本效率.
- 在DR中,标准基线往往无法充分缓解这种高差异,导致训练表现低于最佳.
研究的目的:
- 从理论上推导出DRL中域随机化 (DR) 的无偏差,状态/环境依赖的最佳基线.
- 提出一种新的减差域随机化 (VRDR) 方法,以平衡减差与计算可行性.
- 证明VRDR在加速融合和提高机器人控制政策稳定性和绩效方面的有效性.
主要方法:
- 对DR的最佳基线的理论推导,该基线可以考虑状态和环境的依赖性.
- 开发了减少差异域随机化 (VRDR) 方法,该方法估计了环境子空间内的基线.
- 在6个具有随机动态的机器人控制任务上对VRDR的实证评估.
主要成果:
- 建议的最佳基线理论上保证了与标准常数和状态依赖基线相比,在DR中进一步减少差异.
- 与国家依赖的基线相比,VRDR显示了政策培训趋同的显著加速.
- 经验结果表明VRDR始终实现优越的最终政策,在多个机器人任务中提高了训练稳定性.
结论:
- 新的VRDR方法有效地减少了对DR的政策梯度估计的差异.
- 在随机环境中,VRDR提供了一种实用且有理论依据的方法,以提高DRL培训效率和最终政策质量.
- 这项工作为将DRL应用于需要强有力的概括的现实世界机器人系统提供了重大进展.
相关概念视频
Randomized Experiments
7.0K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
7.0K
Random Variables
12.3K
A random variable is a single numerical value that indicates the outcome of a procedure. The concept of random variables is fundamental to the probability theory and was introduced by a Russian mathematician, Pafnuty Chebyshev, in the mid-nineteenth century.
Uppercase letters such as X or Y denote a random variable. Lowercase letters like x or y denote the value of a random variable. If X is a random variable, then X is written in words, and x is given as a number.
For example, let X = the...
Uppercase letters such as X or Y denote a random variable. Lowercase letters like x or y denote the value of a random variable. If X is a random variable, then X is written in words, and x is given as a number.
For example, let X = the...
12.3K
Variance
9.8K
The deviations show how spread out the data are about the mean. A positive deviation occurs when the data value exceeds the mean, whereas a negative deviation occurs when the data value is less than the mean. If the deviations are added, the sum is always zero. So one cannot simply add the deviations to get the data spread. By squaring the deviations, the numbers are made positive; thus, their sum will also be positive.
The standard deviation measures the spread in the same units as the...
The standard deviation measures the spread in the same units as the...
9.8K
Propagation of Uncertainty from Random Error
703
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
703
Random Sampling Method
11.2K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population. Because measuring the entire population in a study is not practical, researchers use samples to represent the population of interest. Among the various sampling methods used by...
11.2K
Generalization, Discrimination, and Extinction
575
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
575


