对风险敏感的双胞胎分布式批评者具有兰巴达较低的信任度,以持续控制强化学习为准
Onur Osman1, Bahar Yalcin Kavus2, Tolga Kudret Karaca3
1Department of Electric Electronics Engineering, İstanbul Topkapi University, 34087, Istanbul, Turkey.
Scientific reports
|January 30, 2026
概括
带有 λ-Lower Confidence Bound (TDC-λ) 的双胞胎分布式批评者通过使用分布式批评者来控制风险来增强政策之外的强化学习. 这种方法提高了稳定性,减少了连续控制任务的变异,而不会牺牲样本效率.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 强化学习是一种强化学习.
背景情况:
- 政策之外的关键行为体方法,如双延迟深度决定性政策梯度 (TD3),是持续控制强化学习的基础.
- 现有的方法缺乏在时间差异目标中控制风险的明确机制,依赖于标量值估计.
研究的目的:
- 介绍双胞胎分布式批评与λ-Lower Confidence Bound (TDC-λ),这是一个新的算法,用于风险敏感的强化学习.
- 开发一种TD3型算法,该算法包含分布式关键因素和目标选择的风险参数 (λ).
主要方法:
- 实现了一个TD3样式的算法,其中包含两个分布式批评.
- 用较低的信任边界 (μ - λσ) 在关键因素之间制定目标,以管理风险.
- 支持确定性和高斯政策,确定性平均动作用于评估.
主要成果:
- 在标准的MuJoCo基准指标 (HalfCheetah-v4,Hopper-v4,Ant-v4,Walker2d-v4,Humanoid-v4) 上,TDC-λ匹配或改进了最终回报.
- 与强大的基线相比,在任务之间持续减少差异.
- 在高维域上表现出更好的稳定性,风险处罚增加 (较高的λ).
结论:
- 分布式批评结合对风险敏感的目标选择显著提高了非政策强化学习的稳定性.
- 在不影响样本效率的情况下,TDC-λ提供了一种提高稳定性和减少差异的方法.
相关概念视频
01:17Continuous Charge Distributions
8.3K
Imagine a bucket of water. It contains many molecules, of the order of 1026 molecules. Thus, although it contains discrete elements (molecules) at the microscopic level, macroscopically, it can be considered continuous. Small volume elements of water, infinitesimal compared to the bulk of the bucket's volume, still contain many molecules. Under this framework, quantized matter is approximated as continuous for practical purposes.
The electric charge can also be subjected to an analogical...
The electric charge can also be subjected to an analogical...
8.3K
01:22Design Example: Distributing Reinforcements in Concrete Sections
276
The topic explores the practical aspects of adjusting steel reinforcements within a concrete beam section to meet specific design requirements. When designing a reinforced concrete beam, it is essential to distribute the steel reinforcements properly to ensure structural integrity and efficiency. The example provided details a scenario where a beam requires a total steel cross-section of 4 square inches. The engineer identifies that the available steel bars have a nominal diameter of 1.693...
276
01:24Confidence Coefficient
10.6K
The confidence coefficient is also known as the confidence level or degree of confidence. It is the percent expression for the probability, 1-α, that the confidence interval contains the true population parameter assuming that the confidence interval is obtained after sufficient unbiased sampling; for example, if the CL = 90%, then in 90 out of 100 samples the interval estimate will enclose the true population parameter. Here α is the area under the curve, distributed equally under...
10.6K
01:21Confidence Intervals
10.7K
An unbiased point estimate is often insufficient to predict a population estimate, such as population mean or population proportion. In this scenario, a confidence interval is used. A confidence interval is an estimate similar to a sample proportion. However, unlike the point estimate which is a single value, the confidence interval contains a range of values. These values have lower and upper limits, known as confidence limits, and can be designated as L1 and L2, respectively.
A...
A...
10.7K
01:23Reinforcement
918
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
918
01:19Interpretation of Confidence Intervals
10.0K
A confidence interval is a better estimate of the population than a point estimate, as it uses a range of values from a sample instead of a single value.
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
10.0K