On the Value of Myopic Behavior in Policy Reuse

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Leveraging prior knowledge is crucial for efficient learning in unfamiliar situations.
• Reinforcement learning (RL) often requires learning complex policies from scratch, which is data-intensive and time-consuming.
• Reusing existing policies or expert strategies can significantly accelerate RL problem-solving.

Purpose of the Study:

• To introduce a novel framework, Selective Myopic Behavior Control (SMEC), for effective policy reuse in reinforcement learning.
• To demonstrate that short-term behaviors learned from previous tasks are transferable and beneficial for new, unseen tasks.
• To develop a method that adaptively combines prior policy behaviors with task-specific learning for improved decision-making.

Main Methods:

• Developed the Selective Myopic Behavior Control (SMEC) framework.
• Utilized a hybrid value function architecture to evaluate and select behaviors from prior policies.
• Implemented adaptive aggregation of sharable short-term behaviors from prior policies and long-term behaviors of the current task policy.
• Tested the framework on a diverse set of manipulation and locomotion tasks.

Main Results:

• SMEC demonstrated superior performance compared to existing policy reuse methods across various tasks.
• Empirical results validated SMEC's ability to effectively leverage related prior policies.
• The framework showed significant improvements in decision-making through coordinated actions.

Conclusions:

• Short-term behaviors of prior policies are a valuable and sharable resource in reinforcement learning.
• SMEC provides an effective mechanism for adaptive policy reuse, enhancing learning efficiency and performance.
• The proposed approach offers a promising direction for developing more general and adaptable intelligent agents.