Salience Interest Option: Temporal abstraction with salience interest functions

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement Learning (RL) agents learn optimal policies by interacting with environments.
• Temporal abstractions, or options, allow RL agents to make decisions at variable time scales.
• Existing methods like Interest Option Critic (IOC) specialize options but face data inefficiency and inflexibility.

Purpose of the Study:

• To propose a novel approach, Salience Interest Option Critic (SIOC), for more efficient and flexible option discovery in RL.
• To address the limitations of backpropagation-based methods in learning options end-to-end.
• To enhance the value, interpretability, and reusability of learned options.

Main Methods:

• SIOC selects subsets of initiation sets using particle filters, avoiding slow and overfitting backpropagation.
• The method utilizes only reward feedback for rapid and flexible identification of critical subsets.
• Experiments were conducted in both discrete and continuous domains.

Main Results:

• SIOC demonstrated superior efficiency and flexibility compared to existing methods.
• Learned options were more valuable within single tasks.
• Options generated by SIOC showed greater interpretability and reusability in multi-task learning scenarios.

Conclusions:

• SIOC offers a significant advancement in RL option discovery, overcoming key limitations of prior work.
• The particle filter-based approach provides a more robust and adaptable method for learning specialized options.
• SIOC's improvements in efficiency, flexibility, and option reusability have broad implications for RL research and applications.