在深度强化学习中,对政策之外的关键参与者进行相对重要的抽样
Mahammad Humayoo1,2,3, Gengzhong Zheng4, Xiaoqing Dong5
1Hanshan Normal University, Chaozhou, 521041, China. humayoo@hstc.edu.cn.
Scientific reports
|April 24, 2025
概括
相对重要性抽样 (RIS) 通过减少政策外学习的差异来稳定强化学习 (RL). 这种新的方法增强了关键演员算法,提高了基准任务的性能.
科学领域:
- 强化学习是一种强化学习.
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 与政策方法相比,政策之外的强化学习 (RL) 存在不稳定性.
- 目标和行为政策之间的分布不匹配导致高变异和不稳定.
- 重要性抽样 (IS) 减少了分布不匹配,但引入了很高的差异,特别是在连续任务中.
研究的目的:
- 引入相对重要抽样 (RIS) 作为一种减轻差异和稳定政策之外的RL方法.
- 开发第一个没有模型的RIS off-policy actor-critical (RIS-off-PAC) 算法.
- 为了研究光滑参数[公式:见文本]对差异控制的影响.
主要方法:
- 拟议的相对重要抽样 (RIS) 为了平滑重要抽样和减少差异.
- 开发了使用深度神经网络的无模型RIS政策之外的关键行为者 (RIS-off-PAC) 算法.
- 训练有素的演员和批评者网络在奖励函数中使用行为政策行动值.
主要成果:
- 拟议的RIS-off-PAC算法表现出比最先进的RL基准更好的稳定性和性能.
- 绩效是通过OpenAI Gym挑战和合成数据集来评估的.
- 在RIS中的光滑度参数[公式:参见文本]有效控制差异.
结论:
- 相对重要性抽样 (RIS) 提供了一种有希望的方法来稳定和增强政策之外的强化学习.
- 开发的RIS-off-PAC算法为复杂的RL任务提供了强大的和有效的解决方案.
- 这项工作为未来的研究奠定了基础,用于RL的变量减少技术.
相关概念视频
Decision Making: P-value Method
5.2K
The process of hypothesis testing based on the P-value method includes calculating the P- value using the sample data and interpreting it.
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can...
5.2K
Weighted Mean
4.8K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
4.8K
Reinforcement Schedules
115
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
115
Random Sampling Method
10.9K
Sampling is a technique to select a portion (or subset) of the larger population and study that portion (the sample) to gain information about the population. Data are the result of sampling from a population. The sampling method ensures that samples are drawn without bias and accurately represent the population. Because measuring the entire population in a study is not practical, researchers use samples to represent the population of interest. Among the various sampling methods used by...
10.9K
Randomized Experiments
6.6K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
6.6K
Confidence Coefficient
7.4K
The confidence coefficient is also known as the confidence level or degree of confidence. It is the percent expression for the probability, 1-α, that the confidence interval contains the true population parameter assuming that the confidence interval is obtained after sufficient unbiased sampling; for example, if the CL = 90%, then in 90 out of 100 samples the interval estimate will enclose the true population parameter. Here α is the area under the curve, distributed equally under...
7.4K


