在Meta-Reinforcement学习中的上下文表示的全球-本地分解
IEEE transactions on cybernetics
|March 3, 2025
概括
超强化学习 (meta-RL) 代理人使用上下文嵌入适应新任务. 通过将表示分解为全球和本地嵌入,GLOBEX提高了适应性,优于当前的方法.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 机器人技术 机器人技术 机器人技术
背景情况:
- 超强化学习 (meta-RL) 代理人通过从上下文中提取信息来学习适应新任务.
- 有效的上下文表示 (嵌入) 对于新型环境中的元RL代理决策至关重要.
- 现有的meta-RL方法经常使用单个嵌入,这可能无法捕获任务变化的全部范围.
研究的目的:
- 开发一种新的上下文元RL算法,通过分解上下文表示来提高适应性.
- 为了提高嵌入的表达力,对遇到不同的任务分布的元RL代理进行嵌入.
- 研究将全球任务动态与临时局部信号分开的好处.
主要方法:
- 引入了用于上下文元RL (GLOBEX) 的全球本地嵌入,这是一个政策之外的元RL算法.
- 将上下文表示分解为不同的全球和本地嵌入式.
- 采用一种学习过程,最大限度地利用嵌入式中的信息,并使用相互信息约束来解.
主要成果:
- 通过识别全球动态和利用本地信号,GLOBEX有效地适应新任务.
- 拟议的全球-本地嵌入方法捕捉了在上下文空间中更相关的特征.
- 在标准的MuJoCo基准指标上,GLOBEX与最先进的meta-RL算法相比表现优越.
结论:
- 将上下文表示分解为全球和本地嵌入式增强了元RL代理的适应性.
- 格洛贝克斯 (GLOBEX) 提供了一种更具表达性和结构化的方法,用于meta-RL中的上下文表示.
- 这些发现表明,在复杂,动态的环境中改善剂的性能是一个有希望的方向.
相关概念视频
01:24Generalization, Discrimination, and Extinction
399
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
399
01:27Associative Learning
276
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
276
01:15Real-World Application of Classical Conditioning
505
Classical conditioning not only includes the initial pairing of stimuli but also extends to more complex forms, such as higher-order conditioning. Higher-order conditioning involves creating associations beyond the primary conditioned stimulus, resulting in a chain of conditioned responses.
Higher-order, or second-order, conditioning occurs when a neutral stimulus becomes associated with an already established conditioned stimulus through repeated pairings. For instance, if a dog has been...
Higher-order, or second-order, conditioning occurs when a neutral stimulus becomes associated with an already established conditioned stimulus through repeated pairings. For instance, if a dog has been...
505
01:21Cognitive Learning
144
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
144
01:24Reinforcement Schedules
126
Positive reinforcement is a powerful method for teaching new behaviors to both animals and humans. B.F. Skinner demonstrated this with his experiments using rats in a Skinner box. When a rat pressed a lever, it received a food pellet. This immediate reward encouraged the rat to repeat the behavior. This method, where a reward follows every instance of the behavior, is known as continuous reinforcement. It is highly effective for establishing new behaviors quickly.
Once a behavior is learned,...
Once a behavior is learned,...
126
01:22Multi-input and Multi-variable systems
93
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
93