一个简单的统一的不确定性导向框架,用于离线到在线的强化学习
IEEE transactions on neural networks and learning systems
|November 25, 2025
概括
本研究介绍了一个简单的统一不确定性引导 (SUNG) 框架,以改善线下到线上强化学习 (RL). SUNG解决了勘探和分销转移的挑战,在部署之前提高了代理的性能.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 机器人技术 机器人技术 机器人技术
背景情况:
- 离线增强学习 (RL) 可以实现数据驱动的代理培训,但由于数据集质量有限,通常会产生低于最佳的性能.
- 微调在线交互的代理对部署至关重要,但面临着诸如受限制的勘探和分销转移等挑战.
研究的目的:
- 提出一个简单的统一的不确定性引导 (SUNG) 框架,以有效地弥合线下和在线强化学习阶段.
- 解决受约束的探索性行为和国家行动分布转移在线下线上RL中的关键挑战.
主要方法:
- SUNG使用基于变化自编码器 (VAE) 的状态动作访问密度估计器量化不确定性.
- 一个乐观的勘探战略选择具有高价值和不确定性的行动.
- 适应性利用方法平衡了保守的线下RL目标与基于不确定性的标准在线RL目标.
主要成果:
- 在D4RL基准中,SUNG在各种环境和数据集中展示了最先进的在线微调性能.
- 该框架成功地与各种现有的线下RL方法集成.
- 拟议的不确定性量化和指导式勘探/开发策略有效地减轻了线下到线上转移挑战.
结论:
- SUNG框架提供了一个统一而有效的解决方案,通过在线微调来增强线下RL代理.
- 不确定性估计是指导勘探和管理RL分布转移的强大工具.
- 在实际部署之前,SUNG提供了一种实际方法来提高代理性能和可靠性.
相关概念视频
Uncertainty: Overview
1.5K
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
1.5K
Propagation of Uncertainty from Random Error
1.6K
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
1.6K
Reinforcement
797
Positive and negative reinforcement are key concepts in operant conditioning, a learning process where the consequences of a behavior affect the likelihood of that behavior being repeated.
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
Positive reinforcement occurs when a behavior is followed by the presentation of a rewarding stimulus, increasing the frequency of that behavior. For example:
797
Avoidance Learning and Learned Helplessness
2.5K
Avoidance learning and learned helplessness are critical concepts in understanding behavioral responses to negative stimuli.
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
2.5K
Reinforcement Schedules
438
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
438
Observational Learning
802
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
802
