Noise-robust reward machine induction via probabilistic modeling and genetic local search

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) faces challenges in partially observable and noisy environments due to difficulties in inferring latent structures and managing label uncertainty.
• Existing Reward Machine (RM) formalisms struggle with stochastic observations, limiting their application in real-world scenarios.
• Robust policy optimization is hindered by the inherent uncertainty and partial observability in many decision-making tasks.

Purpose of the Study:

• To develop a unified framework for noise-robust Reward Machine (RM) induction and policy optimization in partially observable environments.
• To enhance the interpretability and efficiency of learned reward models for autonomous decision-making.
• To address the fragility of current RM methods when faced with stochastic sensor feedback.

Main Methods:

• Introduced Probabilistic Induction with Genetic Local Search (PI-GLS), integrating probabilistic perception, inductive logic programming, and evolutionary optimization.
• Employed Bayesian inference for uncertain label management and sampling-based symbolic abstraction for noise robustness.
• Developed a belief-aware reward shaping strategy utilizing distributions over RM states to guide policy learning.

Main Results:

• PI-GLS demonstrated substantial improvements in convergence efficiency and robustness to sensor noise compared to existing methods.
• Learned reward models exhibited enhanced interpretability, achieving performance comparable to manually engineered RMs under severe noise.
• The framework proved effective in benchmark domains, showcasing scalability for real-world applications.

Conclusions:

• PI-GLS offers a scalable and effective solution for autonomous decision-making in challenging real-world environments with partial observability and stochastic feedback.
• The proposed methods significantly advance the state-of-the-art in noise-robust RM induction and belief-aware policy learning.
• This work paves the way for more reliable and interpretable AI systems in complex, uncertain settings.